Problem
There's no way to grab an async-profiler flame graph from a running Cassandra node without manually
SSHing in and running the existing flamegraph script by hand. Every other profiling path in this
repo (Pyroscope continuous profiling, the observability stack) is automatic; a one-shot on-demand
capture requires a manual SSH session.
What exists today
packer/cassandra/bin/flamegraph is already baked into every Cassandra AMI:
#!/bin/bash
date=$(date +"%Y-%m-%d_%H-%M-%S")
OUTFILE=/mnt/db1/cassandra/artifacts/flame-$(hostname)-${date}.html
sudo /usr/local/async-profiler/bin/asprof -f $OUTFILE $@ $(cassandra-pid)
echo $OUTFILE
It finds the Cassandra PID, runs asprof with whatever args are passed through (e.g. -e wall,
-e alloc, -d 30), writes an HTML flame graph to /mnt/db1/cassandra/artifacts/, and prints the
output path as its last line of output.
Proposed command
easy-db-lab cassandra asprof [--hosts <list>] -- <asprof args>, e.g.:
easy-db-lab cassandra asprof -e wall
easy-db-lab cassandra asprof --hosts db0,db1 -- -e alloc -d 30
Behavior:
- Resolve target hosts via the standard
HostsMixin (all Cassandra nodes by default, or a subset
via --hosts — same pattern as nt, cql, download-config, etc.).
- On each target host (in parallel — see
HostOperationsService, used elsewhere for
parallel-across-hosts operations), run the existing flamegraph script via
remoteOps.executeRemotely(...), passing through any trailing args untouched (mirrors how
nt/exec run pass through their own trailing args today).
- Capture the script's printed
OUTFILE path (its last stdout line) per host.
- Download each resulting HTML file back to the local machine via
remoteOps.download(...)
(single-file download — see RemoteOperationsService.download(), already used by
download-config for its directory variant) into a local directory (e.g. ./profiles/ in the
cluster workspace, mirroring the existing EASY_DB_LAB_PROFILING/./profiles/ convention used
by the CLI's own async-profiler wrapper in bin/easy-db-lab).
- Print the local path(s) of the downloaded flame graph(s) when done.
Reference implementation patterns already in the codebase
commands/cassandra/DownloadConfig.kt — HostsMixin usage, remoteOps.download* calls,
resolving a per-host remote path and pulling it to a local directory.
commands/cassandra/nt / cql / exec run — trailing-args passthrough to a remote command.
RemoteOperationsService: executeRemotely(), download(), downloadDirectory().
packer/cassandra/bin/flamegraph — the actual remote script this wraps; do not reimplement its
logic (PID lookup, output naming) in Kotlin — just invoke it remotely and fetch the result.
Out of scope
- Changing the
flamegraph script itself (output path/naming convention) unless something about the
new command genuinely requires it.
- Continuous/automatic profiling — this is strictly an on-demand, one-shot capture command. Pyroscope
already covers continuous profiling.
Acceptance Criteria
- WHEN a user runs
easy-db-lab cassandra asprof -e wall with no --hosts, THEN async-profiler runs
on every Cassandra node and an HTML flame graph is downloaded locally per node.
- WHEN
--hosts is passed (e.g. --hosts db0), THEN only that subset of nodes is profiled.
- WHEN the remote
flamegraph script fails on a host (e.g. asprof not installed, no Cassandra PID
found), THEN that host's failure is reported clearly without silently skipping it or aborting the
other hosts' captures.
- WHEN the command completes, THEN the local path of each downloaded flame graph is printed.
Problem
There's no way to grab an async-profiler flame graph from a running Cassandra node without manually
SSHing in and running the existing
flamegraphscript by hand. Every other profiling path in thisrepo (Pyroscope continuous profiling, the observability stack) is automatic; a one-shot on-demand
capture requires a manual SSH session.
What exists today
packer/cassandra/bin/flamegraphis already baked into every Cassandra AMI:It finds the Cassandra PID, runs
asprofwith whatever args are passed through (e.g.-e wall,-e alloc,-d 30), writes an HTML flame graph to/mnt/db1/cassandra/artifacts/, and prints theoutput path as its last line of output.
Proposed command
easy-db-lab cassandra asprof [--hosts <list>] -- <asprof args>, e.g.:Behavior:
HostsMixin(all Cassandra nodes by default, or a subsetvia
--hosts— same pattern asnt,cql,download-config, etc.).HostOperationsService, used elsewhere forparallel-across-hosts operations), run the existing
flamegraphscript viaremoteOps.executeRemotely(...), passing through any trailing args untouched (mirrors hownt/exec runpass through their own trailing args today).OUTFILEpath (its last stdout line) per host.remoteOps.download(...)(single-file download — see
RemoteOperationsService.download(), already used bydownload-configfor its directory variant) into a local directory (e.g../profiles/in thecluster workspace, mirroring the existing
EASY_DB_LAB_PROFILING/./profiles/convention usedby the CLI's own async-profiler wrapper in
bin/easy-db-lab).Reference implementation patterns already in the codebase
commands/cassandra/DownloadConfig.kt—HostsMixinusage,remoteOps.download*calls,resolving a per-host remote path and pulling it to a local directory.
commands/cassandra/nt/cql/exec run— trailing-args passthrough to a remote command.RemoteOperationsService:executeRemotely(),download(),downloadDirectory().packer/cassandra/bin/flamegraph— the actual remote script this wraps; do not reimplement itslogic (PID lookup, output naming) in Kotlin — just invoke it remotely and fetch the result.
Out of scope
flamegraphscript itself (output path/naming convention) unless something about thenew command genuinely requires it.
already covers continuous profiling.
Acceptance Criteria
easy-db-lab cassandra asprof -e wallwith no--hosts, THEN async-profiler runson every Cassandra node and an HTML flame graph is downloaded locally per node.
--hostsis passed (e.g.--hosts db0), THEN only that subset of nodes is profiled.flamegraphscript fails on a host (e.g.asprofnot installed, no Cassandra PIDfound), THEN that host's failure is reported clearly without silently skipping it or aborting the
other hosts' captures.