Skip to content

feat(cassandra): install arbitrary versions/branches onto a live cluster without an AMI rebuild #876

Description

@rustyrazorblade

Problem

Testing an experimental Cassandra build (a git branch, or a tarball from the existing build-cassandra-ref.yml Action, #729) currently requires adding it to cassandra_versions.yaml and baking an entirely new AMI. install_cassandra_version() (packer/cassandra/install/install_cassandra.sh:143) already knows how to install a single version — download tarball, or git-clone+ant-build — but it's only ever invoked from a loop over every entry in /etc/cassandra_versions.yaml, gated by INSTALL_CASSANDRA=1 and run exactly once, at packer bake time (install_cassandra.sh:126-127, :290-300). There's no way to install one additional version onto a node that's already running.

cassandra use <version> (UseCassandra.kt) only switches which pre-baked version is active on a node — it never installs anything.

Scope

In scope:

  • Extract install_cassandra_version() so it can run standalone, against a single version, outside the bake-time loop — invoked remotely via RemoteOperationsService (not ProcessBuilder).
  • New cassandra install <version> subcommand: installs one version onto an already-provisioned, running cluster (all Cassandra nodes, or a --hosts-filtered subset, same mechanism as UseCassandra). Supports both existing install modes: git branch clone+ant-build, and tarball url: download — same schema fields (version, java, url/branch, ant_flags) cassandra_versions.yaml entries already use.
  • Writes the version's entry into /etc/cassandra_versions.yaml on the targeted node(s) before installing — this file is currently baked in once at packer time and only ever read down from a node (Up.kt:630-647), never pushed to; this is the first push-up path. packer/cassandra/bin/set-java-version:27-31 is existing precedent for an in-place yq -i mutation of this file on a live node.
  • lazy: true field on a cassandra_versions.yaml entry: declared/tracked in the repo's cassandra_versions.yaml (so it ships to every node's /etc/cassandra_versions.yaml), but skipped by the packer bake-time install loop — costs nothing at AMI-build time — until cassandra install <version> is explicitly run against a live node.
  • Extend ListVersions.kt (cassandra list) to show lazy-declared-but-not-yet-installed versions, so they're discoverable without already knowing the name from cassandra_versions.yaml.
  • Update openspec/specs/cassandra/spec.md (REQ-CA-001 currently requires multi-version support "on the same AMI" — needs updating/extending for the runtime-install path) and relevant docs in docs/.

Out of scope (decided, do not relitigate):

  • No CI orchestration. cassandra install does not trigger/dispatch build-cassandra-ref.yml and does not poll GitHub Actions for a build result. Run that Action manually, or point cassandra install at a git branch to build directly on the node (existing clone+build path).
  • No persistence across cluster lifecycle. A runtime-installed version is not synced to S3 via ClusterBackupService and does not need to survive down/up or node replacement/addition (not currently supported anyway). If a node is lost, re-run cassandra install.
  • Wiring runtime-install into an AMI rebuild or containerized cluster mode (explicit non-goal already recorded in feat: CI workflow to build an arbitrary Cassandra branch and publish image + tarball #729's design doc).
  • Changes to build-cassandra-ref.yml itself or its fork-support hardening (ci: harden build-cassandra-ref fork support (private forks, tag disambiguation, docs) #765, unrelated).
  • cassandra use auto-installing a missing lazy version — use stays a fast, predictable symlink swap; running it against a not-yet-installed version fails loudly with a "run cassandra install <version> first" message.
  • --force/reinstall-in-place support — re-running cassandra install for an already-installed version is a safe no-op (skip, log "already present"). Can be added later as a fast-follow if reinstalling the same version name (e.g. new commits pushed to the same branch) turns out to be needed.

Acceptance criteria

  • WHEN cassandra install <version> is run with a git repo URL + branch for a version not yet present on the targeted node(s), THEN the branch is cloned and built with ant on that node, the built artifact lands at /usr/local/cassandra/<version>, and the command reports success per targeted host.
  • WHEN cassandra install <version> is run with a tarball url: (e.g. a build-cassandra-ref.yml release asset) for a version not yet present, THEN the tarball is downloaded and extracted to /usr/local/cassandra/<version> on the targeted node(s).
  • WHEN --hosts is passed to filter a subset of Cassandra nodes, THEN only those nodes have /etc/cassandra_versions.yaml updated and the version installed — untargeted nodes are unaffected.
  • WHEN install succeeds on a node, THEN cassandra use <version> on that node works unchanged — no changes needed to use-cassandra or UseCassandra.kt.
  • WHEN cassandra install <version> is run again for a version already installed on a node, THEN it does not error and does not re-download/re-build — safe no-op, logs "already present."
  • WHEN install fails (bad branch name, unreachable/404 tarball URL, ant build failure), THEN the command fails loudly with a clear per-host error identifying the version and the failure reason — no silent partial state, no fallback to another version.
  • WHEN a cassandra_versions.yaml entry has lazy: true and packer bakes a new AMI, THEN the bake-time install loop skips that entry entirely (no download/clone/build time spent), while the entry still appears in the resulting AMI's /etc/cassandra_versions.yaml so cassandra install can read its url/branch/java/ant_flags fields later without the operator re-specifying them.
  • WHEN cassandra use <version> is run for a version that isn't installed on that node, THEN it fails with a clear message directing the operator to run cassandra install <version> first — never a confusing/generic error.
  • WHEN cassandra list is run, THEN lazy-declared-but-not-yet-installed versions are shown, distinguishable from versions actually installed on the node.

Notes / context

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1Priority 1 — highstatus:readyGroomed; awaiting activation

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions