Skip to content

feat(cassandra): add 'cassandra asprof' subcommand to run + fetch async-profiler flame graphs on demand #872

Description

@rustyrazorblade

Problem

There's no way to grab an async-profiler flame graph from a running Cassandra node without manually
SSHing in and running the existing flamegraph script by hand. Every other profiling path in this
repo (Pyroscope continuous profiling, the observability stack) is automatic; a one-shot on-demand
capture requires a manual SSH session.

What exists today

packer/cassandra/bin/flamegraph is already baked into every Cassandra AMI:

#!/bin/bash
date=$(date +"%Y-%m-%d_%H-%M-%S")
OUTFILE=/mnt/db1/cassandra/artifacts/flame-$(hostname)-${date}.html
sudo /usr/local/async-profiler/bin/asprof -f $OUTFILE $@ $(cassandra-pid)
echo $OUTFILE

It finds the Cassandra PID, runs asprof with whatever args are passed through (e.g. -e wall,
-e alloc, -d 30), writes an HTML flame graph to /mnt/db1/cassandra/artifacts/, and prints the
output path as its last line of output.

Proposed command

easy-db-lab cassandra asprof [--hosts <list>] -- <asprof args>, e.g.:

easy-db-lab cassandra asprof -e wall
easy-db-lab cassandra asprof --hosts db0,db1 -- -e alloc -d 30

Behavior:

  1. Resolve target hosts via the standard HostsMixin (all Cassandra nodes by default, or a subset
    via --hosts — same pattern as nt, cql, download-config, etc.).
  2. On each target host (in parallel — see HostOperationsService, used elsewhere for
    parallel-across-hosts operations), run the existing flamegraph script via
    remoteOps.executeRemotely(...), passing through any trailing args untouched (mirrors how
    nt/exec run pass through their own trailing args today).
  3. Capture the script's printed OUTFILE path (its last stdout line) per host.
  4. Download each resulting HTML file back to the local machine via remoteOps.download(...)
    (single-file download — see RemoteOperationsService.download(), already used by
    download-config for its directory variant) into a local directory (e.g. ./profiles/ in the
    cluster workspace, mirroring the existing EASY_DB_LAB_PROFILING/./profiles/ convention used
    by the CLI's own async-profiler wrapper in bin/easy-db-lab).
  5. Print the local path(s) of the downloaded flame graph(s) when done.

Reference implementation patterns already in the codebase

  • commands/cassandra/DownloadConfig.ktHostsMixin usage, remoteOps.download* calls,
    resolving a per-host remote path and pulling it to a local directory.
  • commands/cassandra/nt / cql / exec run — trailing-args passthrough to a remote command.
  • RemoteOperationsService: executeRemotely(), download(), downloadDirectory().
  • packer/cassandra/bin/flamegraph — the actual remote script this wraps; do not reimplement its
    logic (PID lookup, output naming) in Kotlin — just invoke it remotely and fetch the result.

Out of scope

  • Changing the flamegraph script itself (output path/naming convention) unless something about the
    new command genuinely requires it.
  • Continuous/automatic profiling — this is strictly an on-demand, one-shot capture command. Pyroscope
    already covers continuous profiling.

Acceptance Criteria

  • WHEN a user runs easy-db-lab cassandra asprof -e wall with no --hosts, THEN async-profiler runs
    on every Cassandra node and an HTML flame graph is downloaded locally per node.
  • WHEN --hosts is passed (e.g. --hosts db0), THEN only that subset of nodes is profiled.
  • WHEN the remote flamegraph script fails on a host (e.g. asprof not installed, no Cassandra PID
    found), THEN that host's failure is reported clearly without silently skipping it or aborting the
    other hosts' captures.
  • WHEN the command completes, THEN the local path of each downloaded flame graph is printed.

Metadata

Metadata

Labels

enhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions