Skip to content

chore: backport changes from kubefleet repo 6/25 #1149

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zhiying-lin wants to merge 9 commits into from
Closed

chore: backport changes from kubefleet repo 6/25 #1149

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
89c2ca8
chore: upgrade Go version from 1.23.8 to 1.24.4 (#108)
Copilot Jun 16, 2025
25d684b
chore: bump step-security/harden-runner from 2.12.0 to 2.12.1 (#105)
dependabot[bot] Jun 16, 2025
8c83fed
fix: update unexpected error and API Server log error types (#94)
britaniar Jun 17, 2025
03e265d
fix: fix updateRun handling unscheduled bindings (#95)
jwtty Jun 17, 2025
17cf340
test: fix the rollout integration tests flakiness (#110)
zhiying-lin Jun 17, 2025
23615ea
feat: bump version of Karpenter/Azure provider to unblock fix progres…
michaelawyu Jun 24, 2025
385c310
fix: add some delays when creating resourceSnapshot (#97)
zhiying-lin Jun 25, 2025
4ecbe92
Merge remote-tracking branch 'cncf/main' into backport
zhiying-lin Jun 25, 2025
5e9e92e
chore: backport changes from kubefleet repo 6/25
zhiying-lin Jun 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 17 additions & 2 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ on:
paths-ignore: [docs/**, "**.md", "**.mdx", "**.png", "**.jpg"]

env:
GO_VERSION: '1.23.8'
GO_VERSION: '1.24.4'

jobs:
detect-noop:
Expand Down Expand Up @@ -86,6 +86,16 @@ jobs:
HUB_SERVER_URL: 'https://172.19.0.2:6443'

e2e-tests:
strategy:
fail-fast: false
matrix:
customized-settings: [default, custom]
include:
- customized-settings: default
# to shorten the test duration, set the resource snapshot creation interval to 0
resource-snapshot-creation-interval: 0m
- customized-settings: custom
resource-snapshot-creation-interval: 1m
runs-on: ubuntu-latest
needs: [
detect-noop,
Expand Down Expand Up @@ -119,7 +129,11 @@ jobs:

- name: Run e2e tests
run: |
make e2e-tests
if [ "${{ matrix.customized-settings }}" = "default" ]; then
make e2e-tests
else
make e2e-tests-custom
fi
env:
KUBECONFIG: '/home/runner/.kube/config'
HUB_SERVER_URL: 'https://172.19.0.2:6443'
Expand All @@ -129,4 +143,5 @@ jobs:
# TO-DO (chenyu1): to ensure a vendor-neutral experience, switch to a dummy
# property provider once the AKS one is split out.
PROPERTY_PROVIDER: 'azure'
RESOURCE_SNAPSHOT_CREATION_INTERVAL: ${{ matrix.resource-snapshot-creation-interval }}

2 changes: 1 addition & 1 deletion .github/workflows/code-lint.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ on:

env:
# Common versions
GO_VERSION: '1.23.8'
GO_VERSION: '1.24.4'

jobs:

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/codespell.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@0634a2670c59f64b4a01f0f96f84700a4088b9f0 # v2.12.0
uses: step-security/harden-runner@002fdce3c6a235733a90a27c80493a3241e56863 # v2.12.1
with:
egress-policy: audit

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/trivy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ env:
MEMBER_AGENT_IMAGE_NAME: member-agent
REFRESH_TOKEN_IMAGE_NAME: refresh-token

GO_VERSION: '1.23.8'
GO_VERSION: '1.24.4'

jobs:
export-registry:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/upgrade.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ on:
paths-ignore: [docs/**, "**.md", "**.mdx", "**.png", "**.jpg"]

env:
GO_VERSION: '1.23.8'
GO_VERSION: '1.24.4'

jobs:
detect-noop:
Expand Down
2 changes: 1 addition & 1 deletion .golangci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
run:
timeout: 15m
go: '1.23.8'
go: '1.24.4'

linters-settings:
stylecheck:
Expand Down
5 changes: 4 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,10 @@ e2e-tests-v1alpha1: create-kind-cluster run-e2e-v1alpha1

.PHONY: e2e-tests
e2e-tests: setup-clusters
cd ./test/e2e && ginkgo -v -p .
cd ./test/e2e && ginkgo --label-filter="!custom" -v -p .

e2e-tests-custom: setup-clusters
cd ./test/e2e && ginkgo --label-filter="custom" -v -p .

.PHONY: setup-clusters
setup-clusters:
Expand Down
43 changes: 22 additions & 21 deletions charts/hub-agent/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,24 +19,25 @@ _See [helm install](https://helm.sh/docs/helm/helm_install/) for command documen

## Parameters

| Parameter | Description | Default |
|:------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------|
| replicaCount | The number of hub-agent replicas to deploy | `1` |
| image.repository | Image repository | `ghcr.io/azure/azure/fleet/hub-agent` |
| image.pullPolicy | Image pullPolicy | `Always` |
| image.tag | The image release tag to use | `v0.1.0` |
| namespace | Namespace that this Helm chart is installed on | `fleet-system` |
| serviceAccount.create | Whether to create service account | `true` |
| serviceAccount.name | Service account name | `hub-agent-sa` |
| resources | The resource request/limits for the container image | limits: 500m CPU, 1Gi, requests: 100m CPU, 128Mi |
| affinity | The node affinity to use for hubagent pod | `{}` |
| tolerations | The tolerations to use for hubagent pod | `[]` |
| logVerbosity | Log level. Uses V logs (klog) | `5` |
| enableV1Alpha1APIs | If set, the agents will watch for the v1alpha1 APIs. | `false` |
| enableV1Beta1APIs | If set, the agents will watch for the v1beta1 APIs. | `true` |
| hubAPIQPS | QPS to use while talking with fleet-apiserver. Doesn't cover events and node heartbeat apis which rate limiting is controlled by a different set of flags. | `250` |
| hubAPIBurst | Burst to use while talking with fleet-apiserver. Doesn't cover events and node heartbeat apis which rate limiting is controlled by a different set of flags. | `1000` |
| MaxConcurrentClusterPlacement | The max number of clusterResourcePlacement to run concurrently this fleet supports. | `100` |
| ConcurrentResourceChangeSyncs | The number of resourceChange reconcilers that are allowed to run concurrently. | `20` |
| logFileMaxSize | Max size of log file before rotation | `1000000` |
| MaxFleetSizeSupported | The max number of member clusters this fleet supports. | `100` |
| Parameter | Description | Default |
|:-----------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------|
| replicaCount | The number of hub-agent replicas to deploy | `1` |
| image.repository | Image repository | `ghcr.io/azure/azure/fleet/hub-agent` |
| image.pullPolicy | Image pullPolicy | `Always` |
| image.tag | The image release tag to use | `v0.1.0` |
| namespace | Namespace that this Helm chart is installed on | `fleet-system` |
| serviceAccount.create | Whether to create service account | `true` |
| serviceAccount.name | Service account name | `hub-agent-sa` |
| resources | The resource request/limits for the container image | limits: 500m CPU, 1Gi, requests: 100m CPU, 128Mi |
| affinity | The node affinity to use for hubagent pod | `{}` |
| tolerations | The tolerations to use for hubagent pod | `[]` |
| logVerbosity | Log level. Uses V logs (klog) | `5` |
| enableV1Alpha1APIs | If set, the agents will watch for the v1alpha1 APIs. | `false` |
| enableV1Beta1APIs | If set, the agents will watch for the v1beta1 APIs. | `true` |
| hubAPIQPS | QPS to use while talking with fleet-apiserver. Doesn't cover events and node heartbeat apis which rate limiting is controlled by a different set of flags. | `250` |
| hubAPIBurst | Burst to use while talking with fleet-apiserver. Doesn't cover events and node heartbeat apis which rate limiting is controlled by a different set of flags. | `1000` |
| MaxConcurrentClusterPlacement | The max number of clusterResourcePlacement to run concurrently this fleet supports. | `100` |
| ConcurrentResourceChangeSyncs | The number of resourceChange reconcilers that are allowed to run concurrently. | `20` |
| logFileMaxSize | Max size of log file before rotation | `1000000` |
| MaxFleetSizeSupported | The max number of member clusters this fleet supports. | `100` |
| resourceSnapshotCreationInterval | The interval at which resource snapshots are created. | `1m` |
1 change: 1 addition & 0 deletions charts/hub-agent/templates/deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ spec:
- --hub-api-burst={{ .Values.hubAPIBurst }}
- --force-delete-wait-time={{ .Values.forceDeleteWaitTime }}
- --cluster-unhealthy-threshold={{ .Values.clusterUnhealthyThreshold }}
- --resource-snapshot-creation-interval={{ .Values.resourceSnapshotCreationInterval }}
ports:
- name: metrics
containerPort: 8080
Expand Down
2 changes: 2 additions & 0 deletions charts/hub-agent/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ enableGuardRail: true
webhookClientConnectionType: service
forceDeleteWaitTime: 15m0s
clusterUnhealthyThreshold: 3m0s
resourceSnapshotCreationInterval: 1m0s

namespace:
fleet-system

Expand Down
20 changes: 12 additions & 8 deletions cmd/hubagent/options/options.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,8 @@ type Options struct {
PprofPort int
// DenyModifyMemberClusterLabels indicates if the member cluster labels cannot be modified by groups (excluding system:masters)
DenyModifyMemberClusterLabels bool
// ResourceSnapshotCreationInterval is the interval at which resource snapshots are created.
ResourceSnapshotCreationInterval time.Duration
}

// NewOptions builds an empty options.
Expand All @@ -115,14 +117,15 @@ func NewOptions() *Options {
ResourceNamespace: utils.FleetSystemNamespace,
ResourceName: "136224848560.hub.fleet.azure.com",
},
MaxConcurrentClusterPlacement: 10,
ConcurrentResourceChangeSyncs: 1,
MaxFleetSizeSupported: 100,
EnableV1Alpha1APIs: false,
EnableClusterInventoryAPIs: true,
EnableStagedUpdateRunAPIs: true,
EnablePprof: false,
PprofPort: 6065,
MaxConcurrentClusterPlacement: 10,
ConcurrentResourceChangeSyncs: 1,
MaxFleetSizeSupported: 100,
EnableV1Alpha1APIs: false,
EnableClusterInventoryAPIs: true,
EnableStagedUpdateRunAPIs: true,
EnablePprof: false,
PprofPort: 6065,
ResourceSnapshotCreationInterval: 1 * time.Minute,
}
}

Expand Down Expand Up @@ -169,6 +172,7 @@ func (o *Options) AddFlags(flags *flag.FlagSet) {
flags.BoolVar(&o.EnablePprof, "enable-pprof", false, "If set, the pprof profiling is enabled.")
flags.IntVar(&o.PprofPort, "pprof-port", 6065, "The port for pprof profiling.")
flags.BoolVar(&o.DenyModifyMemberClusterLabels, "deny-modify-member-cluster-labels", false, "If set, users not in the system:masters cannot modify member cluster labels.")
flags.DurationVar(&o.ResourceSnapshotCreationInterval, "resource-snapshot-creation-interval", 1*time.Minute, "The interval at which resource snapshots are created.")

o.RateLimiterOpts.AddFlags(flags)
}
17 changes: 9 additions & 8 deletions cmd/hubagent/workload/setup.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,14 +153,15 @@ func SetupControllers(ctx context.Context, wg *sync.WaitGroup, mgr ctrl.Manager,

// Set up a custom controller to reconcile cluster resource placement
crpc := &clusterresourceplacement.Reconciler{
Client: mgr.GetClient(),
Recorder: mgr.GetEventRecorderFor(crpControllerName),
RestMapper: mgr.GetRESTMapper(),
InformerManager: dynamicInformerManager,
ResourceConfig: resourceConfig,
SkippedNamespaces: skippedNamespaces,
Scheme: mgr.GetScheme(),
UncachedReader: mgr.GetAPIReader(),
Client: mgr.GetClient(),
Recorder: mgr.GetEventRecorderFor(crpControllerName),
RestMapper: mgr.GetRESTMapper(),
InformerManager: dynamicInformerManager,
ResourceConfig: resourceConfig,
SkippedNamespaces: skippedNamespaces,
Scheme: mgr.GetScheme(),
UncachedReader: mgr.GetAPIReader(),
ResourceSnapshotCreationInterval: opts.ResourceSnapshotCreationInterval,
}

rateLimiter := options.DefaultControllerRateLimiter(opts.RateLimiterOpts)
Expand Down
2 changes: 1 addition & 1 deletion docker/hub-agent.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Build the hubagent binary
FROM mcr.microsoft.com/oss/go/microsoft/golang:1.23.8 AS builder
FROM mcr.microsoft.com/oss/go/microsoft/golang:1.24.4 AS builder

WORKDIR /workspace
# Copy the Go Modules manifests
Expand Down
2 changes: 1 addition & 1 deletion docker/member-agent.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Build the memberagent binary
FROM mcr.microsoft.com/oss/go/microsoft/golang:1.23.8 AS builder
FROM mcr.microsoft.com/oss/go/microsoft/golang:1.24.4 AS builder

WORKDIR /workspace
# Copy the Go Modules manifests
Expand Down
2 changes: 1 addition & 1 deletion docker/refresh-token.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Build the hubagent binary
FROM mcr.microsoft.com/oss/go/microsoft/golang:1.23.8 AS builder
FROM mcr.microsoft.com/oss/go/microsoft/golang:1.24.4 AS builder

WORKDIR /workspace
# Copy the Go Modules manifests
Expand Down
Loading
Loading