System metrics monitoring (CPU, memory, network I/O) during benchmark runs

docs/reference/callbacks/system_metrics.md

@@ -0,0 +1 @@
+::: llmeter.callbacks.system_metrics

docs/user_guide/callbacks/index.md

@@ -6,6 +6,7 @@ See the dedicated sections in this guide on how [built-in callbacks](../../refer
 
 - [Avoid prompt caching](cache_buster.md) on your target Endpoint
 - [Model and compare the costs](cost.md) of different Endpoints
+- [Monitor system resources](system_metrics.md) (CPU, memory, network I/O) during runs
 - [Track your experiments](mlflow.md) with MLflow
 
 

docs/user_guide/callbacks/system_metrics.md

@@ -0,0 +1,76 @@
+# Monitor System Resources
+
+When running latency benchmarks — especially load tests with high concurrency — it's important to know whether the client machine itself is a bottleneck. High CPU usage, memory pressure, or network saturation on the client side can skew latency measurements without any visible errors.
+
+LLMeter's [`SystemMetricsMonitor`](../../reference/callbacks/system_metrics/#llmeter.callbacks.system_metrics.SystemMetricsMonitor) callback tracks CPU, memory, and network I/O during benchmark runs, providing both real-time visibility through the progress display and persisted statistics in the result.
+
+!!! note "Optional dependency"
+    System metrics monitoring requires `psutil`. Install with:
+    ```bash
+    pip install 'llmeter[system-metrics]'
+    ```
+
+## Quick start
+
+```python
+from llmeter.callbacks.system_metrics import SystemMetricsMonitor
+from llmeter.runner import Runner
+
+monitor = SystemMetricsMonitor(sample_interval=0.5, per_process=True)
+runner = Runner(endpoint=endpoint, callbacks=[monitor], output_path="./results")
+result = await runner.run(payload=payload, clients=10, n_requests=50)
+
+# Aggregated stats are available on the result
+print(f"CPU avg: {result.stats['system_cpu_percent-average']:.1f}%")
+print(f"Memory peak: {result.stats['system_memory_rss_mb-max']:.1f} MB")
+print(f"Network received: {result.stats['system_net_bytes_recv_total']} bytes")
+```
+
+## How it works
+
+The monitor spawns a lightweight background thread that periodically samples system metrics while the benchmark runs:
+
+1. **`before_run`** — resets state, captures the starting network counters, and starts the sampling thread.
+2. **During the run** — the thread collects CPU, memory, and network samples at the configured `sample_interval`.
+3. **`after_run`** — stops the thread and computes aggregated statistics (average, p50, p90, p99, max) that are contributed to `result.stats`.
+
+Because sampling happens on a separate thread, the overhead on benchmark throughput is negligible.
+
+## Configuration
+
+| Parameter | Default | Description |
+| --- | --- | --- |
+| `sample_interval` | `1.0` | Seconds between samples. Lower values yield more granular data. |
+| `per_process` | `True` | If `True`, CPU and memory are scoped to the current Python process. Set to `False` for system-wide monitoring. |
+
+!!! note "Network I/O is always system-wide"
+    Regardless of the `per_process` setting, network I/O metrics are always system-wide because `psutil` does not support per-process network counters on most platforms.
+
+## Live display integration
+
+When attached to a Runner, the monitor surfaces real-time metrics in the progress bar during the run — showing current CPU %, RSS memory, and network receive rate. This happens automatically via the `live_stats()` hook; no extra configuration is needed.
+
+## Reuse across runs
+
+A single `SystemMetricsMonitor` instance can be reused across multiple `runner.run()` calls. The monitor resets its internal state in `before_run`, so each run gets independent samples and statistics.
+
+## Detecting client-side bottlenecks
+
+After a run, check these indicators:
+
+- **`system_cpu_percent-p90` near 100%** — your client process is CPU-bound and may not be able to drive requests fast enough.
+- **`system_memory_rss_mb-max` growing significantly** — memory pressure could trigger GC pauses or swapping.
+- **Network rates plateauing while latency increases** — possible network saturation.
+
+If you observe any of these, consider running LLMeter on a more powerful instance or reducing concurrency.
+
+## Accessing raw samples
+
+For custom analysis (e.g., time-series plots correlating CPU with latency), you can access the raw collected samples:
+
+```python
+for sample in monitor.samples:
+    print(f"t={sample.timestamp:.2f}  cpu={sample.cpu_percent:.1f}%  rss={sample.memory_rss_mb:.1f}MB")
+```
+
+Check out the ["System Metrics Monitoring" example notebook](https://github.com/awslabs/llmeter/blob/main/examples/System%20Metrics%20Monitoring.ipynb) on GitHub for a full walkthrough including load test correlation and time-series analysis.

docs/user_guide/installation.md

@@ -20,6 +20,7 @@ LLMeter also offers optional extra features that require additional dependencies
 - **openai**: Enable testing any endpoint with an [OpenAI](https://platform.openai.com/)-compatible API, including [Amazon Bedrock Mantle](https://docs.aws.amazon.com/bedrock/latest/userguide/bedrock-mantle.html), and LLM gateways like [LiteLLM](https://github.com/BerriAI/litellm), [Kong](https://konghq.com/products/kong-ai-gateway), and [KrakenD](https://www.krakend.io/)
 - **litellm**: Enable testing a range of different models through [LiteLLM](https://github.com/BerriAI/litellm)
 - **mlflow**: Enable logging LLMeter experiments to [MLflow](https://mlflow.org/)
+- **system-metrics**: Enable monitoring CPU, memory, and network I/O during benchmark runs
 
 You can install one or more of these extra options using pip:
 

docs/user_guide/key_concepts.md

@@ -27,3 +27,25 @@ The `Runner` is a low-level API to run a batch of (concurrent) requests through
 An `Experiment` is a high-level, pre-defined analysis to explore a particular aspect of latency or performance - which might run one or more Runs under the hood.
 
 LLMeter's pre-built Experiments are designed based on evaluation best-practices and feedback from our users, but you can always build your own custom Experiments from the lower-level Runner API if needed.
+
+## [Callbacks](../reference/callbacks/index.md): Extensibility hooks
+
+Callbacks let you extend LLMeter's behavior at defined points in the run lifecycle. A callback can run code before/after each request, or before/after an entire run. Built-in callbacks include:
+
+- **[`CostModel`](../reference/callbacks/cost/model.md)**: Estimate costs based on token counts, endpoint time, or custom pricing dimensions.
+- **[`SystemMetricsMonitor`](../reference/callbacks/system_metrics.md)**: Monitor CPU, memory, and network I/O during runs, with live display integration and persisted statistics.
+- **[`MlflowCallback`](../reference/callbacks/mlflow.md)**: Log run parameters and results to MLflow for experiment tracking.
+
+Callbacks are passed to `Runner` or `Experiment` via the `callbacks` parameter:
+
+```python
+from llmeter.callbacks import CostModel
+from llmeter.callbacks.system_metrics import SystemMetricsMonitor
+
+runner = Runner(
+    endpoint=endpoint,
+    callbacks=[SystemMetricsMonitor(), CostModel(...)],
+)
+```
+
+You can also create custom callbacks by subclassing `Callback` — see the [API reference](../reference/callbacks/base.md) for the lifecycle hooks and best practices.

docs/user_guide/metrics.md

@@ -174,6 +174,65 @@ The `CostModel` callback extends `Result.stats` with cost estimates. When attach
 
 See the [Model Costs example notebook](https://github.com/awslabs/llmeter/blob/main/examples/Model%20Costs.ipynb) for a walkthrough of request-based pricing, infrastructure-based pricing, and custom cost dimensions.
 
+### System metrics
+
+The `SystemMetricsMonitor` callback extends `Result.stats` with client-side system resource metrics. When attached to a `Runner` or `Experiment`, it tracks CPU, memory, and network I/O throughout the run.
+
+!!! note
+    Requires the `system-metrics` extra: `pip install 'llmeter[system-metrics]'`
+
+#### Contributed statistics
+
+| Statistic | Description |
+| --- | --- |
+| `system_cpu_percent-average` | Mean CPU usage (%) during the run. |
+| `system_cpu_percent-p50` | Median CPU usage. |
+| `system_cpu_percent-p90` | 90th percentile CPU usage. |
+| `system_cpu_percent-p99` | 99th percentile CPU usage. |
+| `system_memory_rss_mb-average` | Mean resident memory (MB). |
+| `system_memory_rss_mb-max` | Peak resident memory during the run. |
+| `system_memory_rss_mb-p50` / `-p90` / `-p99` | Memory percentiles. |
+| `system_memory_vms_mb-average` / `-max` | Virtual memory statistics. |
+| `system_net_bytes_sent_total` | Total bytes sent over the network during the run. |
+| `system_net_bytes_recv_total` | Total bytes received from the network during the run. |
+| `system_net_bytes_sent_per_second-average` | Average network send rate (bytes/s). |
+| `system_net_bytes_recv_per_second-average` | Average network receive rate (bytes/s). |
+| `system_net_bytes_sent_per_second-p50` / `-p90` / `-p99` | Send rate percentiles. |
+| `system_net_bytes_recv_per_second-p50` / `-p90` / `-p99` | Receive rate percentiles. |
+| `system_samples_collected` | Number of metric samples taken during the run. |
+
+#### Live display
+
+When the `SystemMetricsMonitor` is active, real-time system metrics also appear in the progress bar's live stats display under a **System** column, showing the latest CPU %, memory RSS, and network receive rate.
+
+#### Usage
+
+```python
+from llmeter.callbacks.system_metrics import SystemMetricsMonitor
+from llmeter.runner import Runner
+
+monitor = SystemMetricsMonitor(sample_interval=0.5, per_process=True)
+runner = Runner(endpoint=endpoint, callbacks=[monitor], output_path="./results")
+result = await runner.run(payload=payload)
+
+print(result.stats["system_cpu_percent-average"])
+print(result.stats["system_memory_rss_mb-max"])
+print(result.stats["system_net_bytes_recv_total"])
+```
+
+Key parameters:
+
+- **`sample_interval`** (float, default 1.0): How often to sample metrics, in seconds. Lower values give more granular data but slightly increase overhead.
+- **`per_process`** (bool, default `True`): If `True`, monitors only the current Python process. Set to `False` for system-wide resource tracking (useful when other processes contribute to the workload).
+
+!!! note "Network I/O is always system-wide"
+    Regardless of the `per_process` setting, network I/O metrics are always system-wide because `psutil` does not support per-process network counters on most platforms. If other processes generate significant network traffic during a benchmark, it will be reflected in the network stats.
+
+!!! tip "Detecting client-side bottlenecks"
+    If `system_cpu_percent-p90` approaches 100% or `system_memory_rss_mb-max` grows significantly during high-concurrency load tests, your client machine may be a bottleneck rather than the LLM endpoint. Consider running LLMeter on a more powerful instance or reducing concurrency.
+
+See the [System Metrics Monitoring example notebook](https://github.com/awslabs/llmeter/blob/main/examples/System%20Metrics%20Monitoring.ipynb) for a full walkthrough including load test correlation plots and time-series analysis.
+
 ### Experiment-level metrics
 
 #### Load test