TELEMETRY.md

Performance telemetry

Every AlpaSim run starts Prometheus telemetry by default. No configuration is needed for the basic setup. The wizard allocates ports, starts one Prometheus support service, starts runtime worker /metrics endpoints, and writes the scrape configuration into the run directory.

At the end of the simulation, the runtime queries the Prometheus server and generates metrics_plot.png.

The Prometheus data persists under prometheus/data and can be read by restarting a Prometheus server later.

Central Prometheus discovery

By default, AlpaSim publishes Prometheus file-SD targets under ${defines.filesystem}/prometheus/file-sd. For deploy=local, this resolves under the repo data directory. For deploy=iad, it resolves under the shared IAD Lustre filesystem.

The wizard publishes one central discovery file:

/shared/prometheus/alpasim/<run_uuid>.json

A central Prometheus can discover active AlpaSim runs with:

scrape_configs:
  - job_name: alpasim
    file_sd_configs:
      - files:
          - /shared/prometheus/alpasim/*.json
        refresh_interval: 10s

On Slurm, the process exporter also publishes alpasim_gpu_workload_info{gpu} for every node-local physical GPU assigned to a discovered AlpaSim process. Cross-system dashboards can join this ownership signal to DCGM metrics by run, node, and GPU index.

For normal Docker Compose and Slurm runs, the wizard removes this file when the deployment exits. If a run crashes before cleanup, later AlpaSim startups conservatively remove old discovery files only when they are at least five hours old and all listed targets are unreachable.

To start a local Prometheus and Grafana against a local or mounted file-SD directory, run:

src/tools/scripts/start-prometheus-grafana.sh /shared/prometheus/alpasim

The file-SD directory can also be an SSH path. The script mounts it locally with sshfs:

src/tools/scripts/start-prometheus-grafana.sh \
  <iad-login>:/lustre/fsw/portfolios/av/projects/av_alpamayo_reasoning/data/av_alpamayo_sim/.cache/prometheus/file-sd

The script prints the Prometheus and Grafana URLs.