airflow 2.0.0 — deploy into an existing GVC

[jacobecox]

Review diff: git diff ebe78a1..HEAD -- airflow/versions/2.0.0/
(ebe78a1 is the verbatim cp -R of 1.5.0; everything real is in the commits after it.)

Why a major version

The chart no longer creates a GVC. The gate holds: the in-place helm upgrade from 1.5.0 with 1.5.0's real values file was refused at render before any resource was touched.

Two product changes, both forced by the conversion

1. Single location, by construction. A shared volumeset provisions one volume per location, so two locations would mean two Airflow homes that cannot see each other's DAGs or logs, while the metadata DB and broker are each one volume on one replica. gvc.locations therefore became a single location string, and the default installs on a one-location GVC. The scheduler, dag-processor and triggerer share the webserver container pinned to 1 replica; only Celery workers scale.

2. KEDA is now opt-in, default false — it is a GVC-level setting the chart can no longer set. This is a change from 1.x, and the reason is measured rather than cautious: a metric: keda workload in a KEDA-disabled GVC is accepted silently and sits at minScale — at 0 that is zero replicas, desiredScale: None, no versions and no message. A dead install with no diagnostic surface anywhere.

A latent multi-instance bug, fixed

In 1.5.0 the webserver and Redis carried no options at all, so on a multi-location GVC they would have started one instance per location — N schedulers, N brokers, N split queues. The 0/0 + localOptions pattern now confines every tier, verified live.

A defect testing found, and the control that found it

While running the 403 control, the boot check warned "GVC 'test-gvc' has spec.keda.enabled = false" — while cpln gvc get returned {'enabled': True} at that same moment. The real cause was a missing view grant.

A 403 body is valid, non-empty JSON, so the retry loop broke and the parse succeeded; spec.keda.enabled is absent from an error body and defaults to false, indistinguishable from a genuine false. The location check already defended against this shape; the KEDA check did not.

Fixed by gating both checks on one "the read actually succeeded" flag from the HTTP status, rather than bolting a second emptiness test onto the KEDA branch — an error body can be well-formed enough to survive an emptiness test, and only the status distinguishes it. Verified on both arms:

403 arm:  [airflow] GVC read returned HTTP 403 (attempt 1/3 … 3/3).
          [airflow] WARNING: could not read GVC 'test-gvc' (no 200 response) -- skipping
                    the GVC location and KEDA checks. If this persists, check that the
                    identity has 'view' on the GVC via the chart's GVC policy.
          occurrences of the old false message: 0

200 arm:  [airflow] GVC location check OK -- 'test-gvc' has 'aws-us-east-1'
          no KEDA complaint  -> the fix did not blunt the check it protects

Worth reading: the first attempt at that verification failed to reproduce — the policy was deleted and the workload redeployed immediately, and the read still succeeded. That is the ~4-minute authorization edge-cache. Waiting it out and redeploying produced the 403 arm. A single probe after a policy change is inconclusive in either direction, and here it would have read as "the fix does not work".

Testing

16 rows, 0 outstanding FAIL. Proven on real infrastructure: a DAG executing end to end on a Celery worker; the boot GVC read returning 200 from the real endpoint with the 403 control; both KEDA arms (hard-fail against a KEDA-off GVC, then real autoscaling once enabled); the multi-instance confinement; the wrong-location signal; and the fresh-vs-established asymmetry on both arms.

render-vs-stored 0 unexplained differences — two real backfills (recoveryPolicy: retain on the postgres and redis volume mounts) were declared in the chart, not normalized away in the checker. Drift gate fully Unchanged by the second upgrade, with the gate still clean afterwards.

Probes that settled design questions before build

• localOptions[].autoscaling accepts a keda block, stored verbatim — the only reason KEDA and the 0/0 pattern can coexist.
• target under metric: keda is rejected at apply (target is not allowed when metric is 'keda') and is invisible to helm template.
• The GVC-read bound measured in the pinned image against genuinely slow failures: blackhole 6 s, accept-never-respond 6 s, 3-try loop 24 s, unbounded control still hanging at 25 s. NXDOMAIN returned 0 s and was discarded as proving nothing.

Two things for you

• cpln apply REPLACES a GVC's spec — applying a file containing only spec.keda wiped locationLinks and endpointNamingFormat from test-gvc during the build. Restored and verified byte-for-byte, and now recorded in CLAUDE.md. The safe form is a -o yaml-slim round-trip. This is the opposite of the workload PATCH endpoint, which deep-merges.
• A deliberate gap, not an oversight: if location names a location the GVC lacks, every workload gets zero replicas, so no container runs and the boot check cannot report it. The honest signal is "the release deployed and nothing started", which the README and values comment both state. Closing it would cost an unpinned guard workload for the life of the install.

Untested

The 1.x → 2.0.0 data move. Only tool availability was verified (pg_dump/pg_restore in postgres:18, tar in the airflow image, cpln workload exec --stdin); the README's steps were read against the chart but the piping itself is unproven, and both the README and briefing say so.

Briefing updated.

🤖 Generated with Claude Code