Skip to content

e2e: add upgrade test framework #366

Description

@cardil

Description

Add a test framework for validating operator behavior across disruptive operations on a real cluster -- upgrades first, with chaos and soak testing as future scenarios on the same base. The immediate goal is verifying that existing MCPServer resources survive operator upgrades: version-agnostic, pluggable, with continual (long-running background) checks that validate system behavior during the disruption itself, not just before and after.

Raised in #266. Inspired by knative/pkg/test/upgrade.

Motivation

Operator upgrades are high-risk. Resources created under the previous version must continue to function after the operator is upgraded. This requires verifying:

  • Existing MCPServer resources survive and remain functional
  • Status conditions (Ready, Accepted) are preserved or correctly re-reconciled
  • Backing resources (Deployments, Services, NetworkPolicies) remain intact
  • The system remains available during the upgrade, not just before and after

Unit-level conversion round-trip tests are necessary but insufficient. A real cluster upgrade exercises CRD migration, webhook availability gaps, controller restart sequencing, and other concerns that only surface end-to-end.

Requirements

  • Easy-to-understand DSL for disruptive operations. Test authors declare what to check, not how to schedule it -- upgrades today, chaos injection or soak/long-running stability testing later, on the same primitives.
  • Pluggable. Vendor distributions install and upgrade the operator differently (plain manifests, OLM, a parent operator). They must be able to swap out how install and upgrade work without touching the test DSL itself.
  • Go-first orchestration. The scheduling/wiring logic (what runs when, what runs in parallel, what gates what) is plain Go, testable and readable as Go. Individual install/upgrade steps may shell out to make or other tooling, but the orchestration around them must not.
  • Room for future scenarios. The same base should support chaos and soak tests later without a rewrite, even though only the upgrade scenario is being built now.

Proposed design

The proposed design (types, package layout, scheduling semantics, namespace lifecycle, Installer pluggability) is written up separately: https://gist.github.com/cardil/95bebe35d1ee728830eea21e6c92cc9a

In short: a declarative Suite (install base -> pre-checks -> the disruptive operation and post-checks running concurrently with continual background checks) built on sigs.k8s.io/e2e-framework and native Go testing primitives, with a pluggable Installer so vendor distributions can swap install/upgrade mechanics without touching the test DSL.

Acceptance criteria

  • A reusable, scenario-agnostic test DSL exists for disruptive operations (install base state, run pre-checks, perform the disruption while background checks keep sampling, run post-checks)
  • Background/continual checks cannot prematurely end the sampling window on a single failure, enforced at compile time rather than by convention
  • Installing and performing the disruption are pluggable, so vendor distributions can swap their mechanics without changing the DSL or the test scenarios
  • At least one check validates MCPServer availability throughout an upgrade, not just before and after
  • Upgrade tests reuse the same test-authoring building blocks (assertions/features) as the regular e2e suite where applicable
  • Upgrade tests are opt-in via a dedicated make target and their own Go build tag (not e2e), so the regular suite's build-tag filter can never pull them in; make test-e2e is unaffected and never performs an upgrade mid-run
  • The upgrade scenario's default install/upgrade mechanics guard against false-green results (base and upgraded versions must be verifiably distinct)
  • Vendor distributions can swap install/upgrade orchestration (OLM, a parent operator, etc.) without changing the shared DSL. This is in scope for this issue, not a follow-up. The exact swapping mechanism doesn't need to be nailed down upfront, but should follow the same pluggability pattern being established in fix(e2e): pluggable operator namespace discovery #270 (named/tiered registration, e.g. Register(tier, impl), rather than a raw priority-int list or a file-naming convention) instead of inventing a second, incompatible pattern -- see the design discussion on that PR

Scope notes

  • Downgrade and multi-hop upgrade paths can be added later; not required for v1.
  • The DSL is meant to be reusable for future chaos/soak scenarios; not required for this issue.

/kind feature
/priority important-longterm

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

kind/featureCategorizes issue or PR as related to a new feature.priority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to complete.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions