Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 85 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Cluster Kube Scheduler Operator

A static pod operator that manages the lifecycle of `kube-scheduler` on OpenShift control plane nodes. Built on the [library-go](https://github.com/openshift/library-go) static pod operator framework, it observes cluster configuration and reconciles the target kube-scheduler config into static pod manifests. Installed by the [Cluster Version Operator](https://github.com/openshift/cluster-version-operator) (CVO).

See [ARCHITECTURE.md](ARCHITECTURE.md) for the full design and data flow.

## Build and Test

```bash
make build # Build all binaries (operator + OTE test runner)
make test # Unit tests (./pkg/... ./cmd/... ./bindata/...)
make verify # Formatting, vetting, golang version checks
make e2e # E2E operator tests (1h timeout)
make test-e2e-preferred-host # E2E preferred-host tests (1h timeout)
```

Go version: see `go.mod`.

## Project Structure

| Directory | Purpose |
|-----------|---------|
| `bindata/` | Embedded assets: default config, static pod template, alerts, RBAC, bootstrap manifests |
| `cmd/cluster-kube-scheduler-operator/` | Operator binary entry point (operator, installer, pruner) |
| `cmd/cluster-kube-scheduler-operator-tests-ext/` | OpenShift Tests Extension (OTE) test runner entry point |
| `cmd/render/` | Bootstrap manifest renderer for cluster installation |
| `manifests/` | CVO deployment manifests (namespace, deployment, RBAC, ServiceMonitors) |
| `pkg/operator/configmetrics/` | Prometheus metrics derived from the Scheduler CR |
| `pkg/operator/configobservation/` | Configuration observers — watch cluster state to produce operand config |
| `pkg/operator/operatorclient/` | Namespace constants and operator client interfaces |
| `pkg/operator/resourcesynccontroller/` | Syncs ConfigMaps/Secrets between namespaces |
| `pkg/operator/starter.go` | Operator initialization — creates clients, informers, and starts all controllers |
| `pkg/operator/targetconfigcontroller/` | Renders observed config + defaults into kube-scheduler ConfigMaps |
| `test/e2e/` | E2E test suite (operator behavior) |
| `test/e2e-preferred-host/` | E2E test suite (preferred-host kube-apiserver communication) |
| `test/library/` | Shared test utilities |

## Controller Pattern

Controllers use the library-go `factory.Controller` base. Each controller has a `sync(ctx, syncContext)` method called by the framework on informer events or periodic resyncs. The operator wires them in `pkg/operator/starter.go` via `RunOperator()`.

Config observers follow a specific pattern: each observer function receives the existing config and returns `(observedConfig, errors)`. Observers are registered in `pkg/operator/configobservation/configobservercontroller/observe_config_controller.go`.

## Key Conventions

- **Namespaces:** `openshift-kube-scheduler-operator` (operator), `openshift-kube-scheduler` (operand), `openshift-config` (user config), `openshift-config-managed` (platform config). Constants in `pkg/operator/operatorclient/interfaces.go`.
- **Logging:** `k8s.io/klog/v2` with verbosity levels
- **Error handling:** wrap with `fmt.Errorf("context: %w", err)`
- **Feature gates:** controllers that depend on feature gates use `FeatureGateAccessor` from library-go; wait for gates before starting

## What Not To Do

These are specific mistakes that are easy to make in this codebase — read before changing anything:

- **Don't modify `vendor/` directly** — use `go mod tidy && go mod vendor`. Manual edits to `vendor/` will be overwritten and break reproducibility.

- **Don't edit `bindata/assets.go`** — this file uses Go's `embed` directive and is auto-generated. Update the asset files under `bindata/assets/` and `bindata/bootkube/`; never edit the embed wrapper itself.

- **Don't add a config observer without registering it** — new observer functions must be added to the observer list in `pkg/operator/configobservation/configobservercontroller/observe_config_controller.go`. A function that is never registered will silently do nothing.

- **Don't bypass the static pod upgrade protocol** — the installer/revision/pruner controllers in library-go manage rollout ordering and availability guarantees. Do not apply static pod changes directly to nodes or skip the revision mechanism.

- **Don't use direct API calls in config observers** — use the listers available in `configobservation.Listers`. Direct `client.Get()` or `client.List()` calls in observer functions add latency and load, and break the informer-driven design.

- **Don't fix library-go bugs here** — if a bug is in library-go, fix it upstream in [library-go](https://github.com/openshift/library-go) and vendor the fix. Workarounds in this repo create divergence that makes future rebases harder.

- **Don't rename namespace constants** without a full grep of all usages — `OperatorNamespace`, `TargetNamespace`, etc. in `pkg/operator/operatorclient/interfaces.go` are used throughout the codebase and in static asset templates under `bindata/`.

- **Don't assume feature gates are available at startup** — `featureGateAccessor.InitialFeatureGatesObserved()` must be awaited (with a timeout) before accessing feature gates. The operator already does this in `starter.go`; don't bypass or reorder it.

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for full guidelines. Key rules:

- Do not modify files under `vendor/`. Use `go mod tidy && go mod vendor`.
- `bindata/assets.go` uses Go's `embed` directive to embed asset files — update the embedded files, not this file.
- Write unit tests for every change. E2E tests for significant features.
- Backwards compatibility matters — deprecate before removing.
- Before modifying the operator API, ensure there is a corresponding enhancement proposal in [openshift/enhancements](https://github.com/openshift/enhancements). API changes require design review and approval.

## Testing

- **Unit tests:** co-located `*_test.go` files, table-driven, `go test ./pkg/... ./cmd/... ./bindata/...`
- **E2E tests:** suites under `test/e2e/` and `test/e2e-preferred-host/`, each with its own Makefile target. Tests use the Ginkgo v2 framework via the OTE framework.
- **OTE framework:** `cluster-kube-scheduler-operator-tests-ext` binary. See [CONTRIBUTING.md](CONTRIBUTING.md#openshift-tests-extension-ote) for usage.
110 changes: 110 additions & 0 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# Architecture

## Overview

The cluster-kube-scheduler-operator is a static pod operator that manages the `kube-scheduler` on OpenShift control plane nodes. It is deployed by the Cluster Version Operator (CVO) and uses the [library-go](https://github.com/openshift/library-go) static pod operator framework.

The operator's primary responsibilities:
- Observe cluster configuration from multiple sources and synthesize kube-scheduler config
- Manage kube-scheduler static pods across control plane nodes (install, revision, prune)
- Report status via the `ClusterOperator/kube-scheduler` resource

## Data Flow

```text
config.openshift.io resources
(Scheduler, APIServer)
┌───────────────────────────────────────────────────┐
│ Config Observer Controllers │
│ (observe external state, produce observedConfig) │
└──────────────────────┬────────────────────────────┘
│ observedConfig (sparse JSON)
┌───────────────────────────────────────────────────┐
│ Target Config Controller │
│ (merge defaults + observedConfig + overrides │
│ → render ConfigMaps in target namespace) │
└──────────────────────┬────────────────────────────┘
│ ConfigMaps
┌───────────────────────────────────────────────────┐
│ Static Pod Controllers (library-go) │
│ Installer → Revision Controller → Pruner │
│ (roll out new revisions to each control plane │
│ node as static pod manifests) │
└──────────────────────┬────────────────────────────┘
kube-scheduler static pods
(one per control plane node)
```

## Operator Startup

Entry point: `cmd/cluster-kube-scheduler-operator/main.go` → `pkg/operator/starter.go:RunOperator()`.

Startup sequence:
1. Create clients (Kubernetes, config, operator)
2. Create informers for watched namespaces (see [Namespaces](#namespaces))
3. Initialize feature gates via `FeatureGateAccessor` and wait for observation (1-minute timeout)
4. Create and start all controllers concurrently
5. Block until context cancellation

## Namespaces

| Namespace | Constant | Purpose |
|-----------|----------|---------|
| `openshift-config` | `GlobalUserSpecifiedConfigNamespace` | User-provided configuration (policy configmaps) |
| `openshift-config-managed` | `GlobalMachineSpecifiedConfigNamespace` | Platform-managed configuration |
| `openshift-kube-scheduler-operator` | `OperatorNamespace` | Operator deployment and its resources |
| `openshift-kube-scheduler` | `TargetNamespace` | Operand: kube-scheduler pods and config |

The `ResourceSyncController` copies ConfigMaps and Secrets between these namespaces as needed (e.g. policy configmap from `openshift-config` to the target namespace).

## Static Pod Management

The operator uses library-go's `staticpod.NewBuilder()` to manage kube-scheduler static pods. This framework provides:

- **Installer controller** — creates new static pod revisions on each control plane node. Uses a custom installer command (`cluster-kube-scheduler-operator installer`).
- **Revision controller** — tracks revisions of ConfigMaps and Secrets. When any revisioned resource changes, a new revision is created. The first ConfigMap in the list (`kube-scheduler-pod`) contains the static pod manifest template.
- **Pruner** — removes old static pod revisions to free disk space.
- **PDB guard** — ensures availability during upgrades (only on multi-node clusters; disabled for single-node).

Resources are split into two categories:
- **Revisioned** — ConfigMaps and Secrets declared in `deploymentConfigMaps`/`deploymentSecrets` in `starter.go`. Any change to these triggers a new static pod revision. The first ConfigMap in the list contains the static pod manifest template.
- **Unrevisioned certs** — Secrets declared in `CertSecrets` in `starter.go`, managed by `WithUnrevisionedCerts`. These are updated in-place without triggering a new revision.

## Configuration Observers

Configuration observers watch external cluster resources and produce a sparse JSON config (`observedConfig`) that gets merged into the kube-scheduler configuration. Each observer function receives the existing config and returns `(observedConfig, errors)`.

Observers are registered in `pkg/operator/configobservation/configobservercontroller/observe_config_controller.go`:

| Package | Watches | Effect |
|---------|---------|--------|
| `scheduler/` | `Scheduler` CR | Syncs `policy-configmap` from `openshift-config` to `openshift-kube-scheduler` if `.spec.policy.name` is set |
| *(library-go)* `apiserver` | `APIServer` CR | TLS security profile |

## Target Config Controller

`pkg/operator/targetconfigcontroller/` takes the merged configuration (defaults + observedConfig + unsupportedConfigOverrides) and renders it into concrete resources in the target namespace:

- `config` ConfigMap — the main kube-scheduler configuration, built by merging `bindata/assets/config/defaultconfig.yaml` with the observed config and applying the scheduling profile
- `kube-scheduler-pod` ConfigMap — the static pod manifest template

The target config controller handles scheduling profiles (`LowNodeUtilization`, `HighNodeUtilization`, `NoScoring`) and passes cluster feature gates as `--feature-gates` flags to the scheduler binary.

## Render Command

`cmd/render/` is a bootstrap manifest renderer used during cluster installation. It takes installer-provided inputs and renders the initial set of manifests needed to bootstrap kube-scheduler before the operator is running. The templates live in `bindata/bootkube/`.

## Other Controllers

| Controller | Purpose |
|-----------|---------|
| `StaticResourceController` | Applies static manifests from `bindata/` (namespace, network policies, RBAC, service, ServiceAccount, kubeconfig) |
| `ClusterOperatorStatus` | Reports operator status, versions, and related objects to `ClusterOperator/kube-scheduler` |
| `ResourceSyncController` | Syncs ConfigMaps/Secrets between namespaces (e.g. policy configmap) |
| `ConfigMetrics` | Registers Prometheus metrics derived from the `Scheduler` CR (`cluster_master_schedulable`, `cluster_legacy_scheduler_policy`, `cluster_default_node_selector`) |
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
@AGENTS.md
Loading