Skip to content
This repository was archived by the owner on Jan 13, 2025. It is now read-only.

[WIP] Implement HVPA v2 proposal #88

Open
amshuman-kr wants to merge 4 commits into
base: master
Choose a base branch
from
Open

[WIP] Implement HVPA v2 proposal #88

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
325a8f2
HVPA redesign (#83)
Nov 30, 2020
bb2d342
Mock controller-runtime client for unit tests.
Aug 17, 2021
87de059
Migrate unit tests to use gmock.
Aug 21, 2021
0c8d734
Retain v1alpha1 and move new changes to v1alpha2
Sep 10, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
292 changes: 9 additions & 283 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,10 @@
1. HPA and VPA are recommended to be mixed only in case HPA is used for custom/external metrics as mentioned [here](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler#limitations-of-beta-version). But in some scenarios it might make sense to use them both even for CPU and memory (e.g. `kube-apiserver` or ingress etc.)
1. VPA updates the pods directly (via webhooks) which may confuse HPA (especially, when used for the same scaling metrics) as it may not see the changes in the upstream `targetRefs`.
1. Where there is functionality missing in either HPA or VPA, introduce them to provide more flexibility during horizontal and vertical scaling. Especially, for the components that experience disruption during scaling.
1. [Weight-based scaling](#weight-based-scaling) horizontally and vertically silmultaneously.
1. Support for configurable (at the resource level) threshold levels to trigger VPA (and possibly HPA) updates to minimize unnecessary scaling of components. Especially, if scaling is disruptive.
1. Support for configurable (at the resource level) stabilisation window in all the four directions (up/down/out/in) to stabilize scaling of components. Especially, if scaling is disruptive.
1. Support for configurable maintenance window for scaling (especially, scaling in/down) for components that do not scale well smoothly (mainly, `etcd`, but to a lesser extent `kube-apiserver` as well for `WATCH` requests). This could be as an alternative or complementary to the stabilisation window mentioned above.
1. Support for flexible update policy for all four scaling directions (Off/Auto/ScaleUp). ScaleUp would only apply scale up and not scale down (vertically or horizontally). This is again from the perspective of components which experience disruption while scaling (mainly, `etcd`, but to a lesser extent `kube-apiserver` as well for `WATCH` requests). For such components, a `ScaleUp` update policy will ensure that the component can scale up (with some disruption) automatically to meet the workload requirement but not scale down to avoid unnecessary disruption. This would mean over-provisioning for workloads that experience a short upsurge.
1. Horizontally and vertically silmultaneously
1. Support for configurable (at the resource level) threshold levels to trigger VPA (and possibly HPA) updates to minimize unnecessary scaling of components. Especially, if scaling is disruptive
1. Support for configurable (at the resource level) stabilisation time window in both the directions (up/down) to stabilize scaling of components. Especially, if scaling is disruptive
1. Support for configurable maintenance window for scaling (especially, scaling in/down) for components that do not scale well smoothly (mainly, `etcd`, but to a lesser extent `kube-apiserver` as well for `WATCH` requests). This could be as an alternative or complementary to the stabilisation window mentioned above
1. Custom resource support for VPA updates.
* At the beginning, this would be with some well-defined annotations to describe which part of the CRD to update the resource requests.
* But eventually, we can think of proposing a `Resource` subresource (per container) along the lines of `Scale` subresource.
Expand All @@ -25,289 +24,16 @@

* It is *not* a goal of HVPA to duplicate the functionality of the upstream components like [HPA](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscaler/) and [VPA](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler).


### Weight based scaling
In this mode, deployment's scaling is divided into stages depending on number of replicas. A user can provide an array of intervals `x1` to `x2` and corresponding `vpaWeight`s between and including `0` and `1`. `x1` and `x2` are the number of replicas where that stage begins and ends respectively. In a particular stage, HVPA controller will consider VPA's recommendations according to the weights provided, and scale the deployment accordingly. The weights given to the HPA's recommendation will be `1 - vpaWeight`.

According to VPA, new `requests` will be `currentRequest + (targetRequest - currentRequest) * vpaWeight`

According to HPA, new number of replicas will be:
* ceil of `currentReplicas + (desiredReplicas - currentReplicas) * (1 - vpaWeight)` if desired replicas is more than current replicas
* floor of `currentReplicas + (desiredReplicas - currentReplicas) * (1 - vpaWeight)` if desired replicas is less than current replicas

`vpaWeight` = 1 will result in pure vertical scaling
`vpaWeight` = 0 will result in pure horizontal scaling

```
Resource request vs number of replicas curve could typically look like:
Here First stage Vpa weight = 0, second stage Vpa weight is a fraction, third stage Vpa weight = 0

resource
request
^
|
| ____________
| /|
| /
| / |
| /
| ___________/ |
|
|-------------|----|-----------|----->
min x1 x2 max #Replicas
```

```
Example of vpaWeight vs number of replicas.
Here first stage Vpa weight = 0, second stage Vpa weight = 0.4, third stage Vpa weight = 0

vpaWeight
^
|
0.4| |-------|
| | |
0| ________| |____________
|------------------------------------->
x1 x2 #Replicas
```

**Scaling of `limits`**:

HVPA will scale `limits` also along with `requests` based on following criteria:
1. If originalLimit is not set, don't set limit.
1. If originalLimit is set but originalRequest not set - K8s will treat the pod as if they were equal, then set limit equal to request
1. If originalLimit and originalRequest are set and if they are equal, recommend limit equal to request.
1. If limit scaling parameters are not set in `hvpa.spec.vpa` then scale the limits proportionaly as done by VPA
1. If the scaling parameters are provided, then choose the max of the 2 possible values in `hvpa.spec.vpa.limitsRequestsGapScaleParams` (percentage and value)

#### Pros
* Works even if HPA and VPA act on different metrices
* Gives better control to user on scaling of apps
#### Cons
* Need to define behavior when:
* current number of replicas is `x2`, the deployment has not yet scaled up to `maxAllowed`, and user has chosen to scale only horizontally after `x2`. In this case, the deployment cannot scale up vertically anymore if the load increases.
* current number of replicas is `x1`, the deployment has not yet scaled down to `minAllowed`, and user has chosen to scale only horizontally for lower values than `x1`. In this case, the deployment cannot scale down vertically anymore if the load decreases.
* Will need both VPA and HPA to be deployed in a recommendation-only mode.

#### Mitigation
* `x1` and `x2` will be optional, If either or both of them are not provided, vertical scaling will be done until `minAllowed` and/or `maxAllowed` correspondingly.
* Once replica count reaches `maxReplicas`, and VPA still recommends higher resource requirements, then vertical scaling will be done unconditionally

#### Spec
[Here](#types) is the spec for HVPA controller

Example 1:

```yaml
apiVersion: autoscaling.k8s.io/v1alpha1
kind: Hvpa
metadata:
name: hvpa-sample
spec:
maintenanceTimeWindow:
begin: "220000-0000"
end: "230000-0000"
hpa:
deploy: true
selector:
matchLabels:
key1: value1
scaleUp:
updatePolicy:
updateMode: "Auto"
stabilizationDuration: "2m"
scaleDown:
updatePolicy:
updateMode: "Auto"
stabilizationDuration: "3m"
template:
metadata:
labels:
key1: value1
spec:
maxReplicas: 10
minReplicas: 1
metrics:
- resource:
name: memory
targetAverageUtilization: 50
type: Resource
- resource:
name: cpu
targetAverageUtilization: 50
type: Resource
vpa:
deploy: true
selector:
matchLabels:
key2: value2
scaleUp:
updatePolicy:
updateMode: "Auto"
stabilizationDuration: "2m"
minChange:
cpu:
value: "3"
percentage: 80
memory:
value: "3"
percentage: 80
scaleDown:
updatePolicy:
updateMode: "MaintenanceWindow"
stabilizationDuration: "3m"
minChange:
cpu:
value: "3"
percentage: 80
memory:
value: "3"
percentage: 80
limitsRequestsGapScaleParams:
cpu:
percentage: 30
value: "1"
memory:
percentage: 30
value: 1G
template:
metadata:
labels:
key2: value2
spec:
resourcePolicy:
containerPolicies:
- containerName: resource-consumer
minAllowed:
memory: 400Mi
maxAllowed:
memory: 3000Mi
weightBasedScalingIntervals:
- vpaWeight: 0
startReplicaCount: 1
lastReplicaCount: 2
- vpaWeight: 0.6
startReplicaCount: 3
lastReplicaCount: 7
- vpaWeight: 0
startReplicaCount: 7
lastReplicaCount: 10
targetRef:
apiVersion: apps/v1
kind: Deployment
name: resource-consumer
```
```
resource
request
^
|
| _________
| /|
| /
| / |
| /
| ___________/ |
|
|-------------|----|--------|----->
1 3 7 10 #Replicas
```


Example 2:
```yaml
apiVersion: autoscaling.k8s.io/v1alpha1
kind: Hvpa
metadata:
name: hvpa-sample
spec:
weightBasedScalingIntervals:
- vpaWeight: 1
startReplicaCount: 1
lastReplicaCount: 1
- vpaWeight: 0.6
startReplicaCount: 2
lastReplicaCount: 4
- vpaWeight: 0
startReplicaCount: 5
lastReplicaCount: 10
hpa:
.
.
.
```
```
resource
request
^
| ______
| /
| /
| /
| /
| |
| |
| |
|
|--|----|-----|------->
1 4 10 #Replicas
```


Example 3: After maxReplicas, even if weight is not mentioned, there is only vertical scaling because HPA is limited by maxReplicas
```yaml
apiVersion: autoscaling.k8s.io/v1alpha1
kind: Hvpa
metadata:
name: hvpa-sample
spec:
weightBasedScalingIntervals:
- vpaWeight: 0
startReplicaCount: 1
lastReplicaCount: 3
- vpaWeight: 0.6
startReplicaCount: 4
lastReplicaCount: 10
hpa:
.
.
template:
.
.
spec:
minReplicas: 1
maxReplicas: 10
metrics:
.
.
.
.
.
```
```
resource
request
^
|
| |
| |
| .
| /
| /
| /
| /
| ____/
|
|--|---|----|------>
1 3 10 #Replicas
```
### Design of HVPA Controller
See the design documentation in the `./docs/` [here](docs/proposals/HVPA-EP-0001/HVPA-EP-001.md).

### Custom Resource Definition
:construction: Work in progress

#### Types

See [here](https://github.com/gardener/hvpa-controller/blob/a336e86111bbd5ec87f5a749003ba39b4735261e/api/v1alpha1/hvpa_types.go#L28-L305).
[DEPRECATED] See [here](https://github.com/gardener/hvpa-controller/blob/a336e86111bbd5ec87f5a749003ba39b4735261e/api/v1alpha1/hvpa_types.go#L28-L305).

#### Example

See [here](https://github.com/gardener/hvpa-controller/blob/a698b08bc371525f666bb2f4766ccb9a403a0d4b/config/samples/autoscaling_v1alpha1_hvpa.yaml).
[DEPRECATED] See [here](https://github.com/gardener/hvpa-controller/blob/a698b08bc371525f666bb2f4766ccb9a403a0d4b/config/samples/autoscaling_v1alpha1_hvpa.yaml).
14 changes: 8 additions & 6 deletions ...troller/api/v1alpha1/groupversion_info.go → api/v1alpha2/groupversion_info.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,10 @@ See the License for the specific language governing permissions and
limitations under the License.
*/

// Package v1alpha1 contains API Schema definitions for the autoscaling v1alpha1 API group
// Package v1alpha2 contains API Schema definitions for the autoscaling v1alpha2 API group
// +kubebuilder:object:generate=true
// +groupName=autoscaling.k8s.io
package v1alpha1
// +groupName=autoscaling.gardener.cloud
package v1alpha2

import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
Expand All @@ -28,13 +28,15 @@ import (

var (
// GroupName is the group name use in this package
GroupName = "autoscaling.k8s.io"
GroupName = "autoscaling.gardener.cloud"

// GroupNameVpa is the group name used to register these objects
GroupNameVpa = "autoscaling.k8s.io"
// SchemeGroupVersionVpa is group version used to register these objects
SchemeGroupVersionVpa = schema.GroupVersion{Group: GroupName, Version: "v1beta2"}
SchemeGroupVersionVpa = schema.GroupVersion{Group: GroupNameVpa, Version: "v1beta2"}

// SchemeGroupVersionHvpa is group version used to register these objects
SchemeGroupVersionHvpa = schema.GroupVersion{Group: GroupName, Version: "v1alpha1"}
SchemeGroupVersionHvpa = schema.GroupVersion{Group: GroupName, Version: "v1alpha2"}

// SchemeBuilder is used to add go types to the GroupVersionKind scheme
SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes)
Expand Down
Loading