Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/reference/s3fs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# s3fs

::: llmeter.s3fs
16 changes: 16 additions & 0 deletions docs/user_guide/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,3 +52,19 @@ LLMeter measures **end-to-end latencies** and uses pure-Python ([asyncio](https:
> If hosting LLMeter on [burstable](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-performance-instances.html) Cloud instance types (e.g. AWS `t3`, `t4g`, etc), be aware that sustained network and compute limits are lower than short-term burst resources.

Because generative AI inference is a compute-intensive task, load testing LLMs has not typically required very high-powered (compute, network) resources from the client side. This led us to a deliberate decision to keep LLMeter simple to install and use by avoiding more complex distributed computing frameworks, in favour of plain Python+asyncio. If you have a use-case with such a fast LLM or large request volume that this approach is limiting to you - please share your feedback!

## S3 Storage Support

LLMeter includes built-in support for Amazon S3 paths (`s3://`) with no extra dependencies. The `boto3` package (already a core dependency) provides the S3 client, and LLMeter's built-in filesystem backend handles the rest.

No additional installation is needed — just ensure your AWS credentials are configured (via environment variables, `~/.aws/credentials`, or an IAM role).

```python
from upath import UPath as Path

# S3 paths work out of the box
results = Path("s3://my-bucket/experiments/run-001/metrics.json")
data = results.read_text()
```

See the [S3 Storage guide](s3_storage.md) for details on configuration, async usage, and fallback behavior.
97 changes: 97 additions & 0 deletions docs/user_guide/s3_storage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# S3 Storage

LLMeter supports reading and writing experiment results to Amazon S3 using standard path operations. This works transparently through the [UPath](https://github.com/fsspec/universal_pathlib) integration — you can use `s3://` paths anywhere you'd use local file paths.

## How It Works

LLMeter includes a built-in S3 filesystem backend (`Boto3S3FileSystem`) that uses `boto3` directly. This replaces the previous `s3fs`/`aiobotocore` dependency chain, eliminating the version conflicts those packages caused with `boto3`.

The backend registers itself automatically when you import `llmeter`. No extra installation or configuration is needed beyond having valid AWS credentials.

```python
from upath import UPath as Path

# These just work — no extra packages needed
results_path = Path("s3://my-bucket/experiments/run-001/")
results_path.mkdir(parents=True, exist_ok=True)

# Write results
(results_path / "metrics.json").write_text('{"latency_p50": 0.42}')

# Read results
data = (results_path / "metrics.json").read_text()

# List files
for p in results_path.iterdir():
print(p.name)

# Glob patterns
json_files = list(results_path.glob("**/*.json"))
```

## Credentials

The backend uses standard AWS credential resolution — the same mechanisms `boto3` uses:

1. Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)
2. AWS config files (`~/.aws/credentials`, `~/.aws/config`)
3. IAM instance profiles (on EC2/ECS/Lambda)
4. SSO/credential process configurations

If `boto3` works in your environment, S3 paths in LLMeter will too.

## Custom Configuration

For advanced use cases, you can instantiate the filesystem directly:

```python
import boto3
from llmeter.s3fs import Boto3S3FileSystem

# Custom session (e.g., specific profile)
session = boto3.Session(profile_name="my-profile")
fs = Boto3S3FileSystem(session=session)

# Custom region
fs = Boto3S3FileSystem(region_name="eu-west-1")

# S3-compatible services (MinIO, LocalStack)
fs = Boto3S3FileSystem(endpoint_url="http://localhost:9000")
```

## Fallback Behavior

If you prefer a different S3 backend, the built-in one steps aside:

- If `s3fs` is installed, LLMeter defers to it
- If `obstore` is installed, LLMeter defers to it
- Otherwise, LLMeter's built-in `Boto3S3FileSystem` handles S3 paths

This means existing setups with `s3fs` continue to work unchanged.

## Limitations

- **File size**: The backend uses `put_object` for writes, which has a 5 GiB limit. This is fine for LLMeter's typical use case (JSON/JSONL result files in the KB–MB range).
- **Append mode**: `open(mode="a")` uses read-modify-write semantics. Concurrent writers to the same key may cause data loss.
- **No multipart upload**: Files larger than 5 GiB require multipart upload, which is not implemented.

## Async Support

The backend is async-native, wrapping synchronous boto3 calls in `asyncio.to_thread()` for non-blocking execution. This means it works well in async code:

```python
import asyncio
from llmeter.s3fs import Boto3S3FileSystem

async def main():
fs = Boto3S3FileSystem()

# Concurrent reads
results = await asyncio.gather(
fs._cat_file("my-bucket/file1.json"),
fs._cat_file("my-bucket/file2.json"),
fs._cat_file("my-bucket/file3.json"),
)

asyncio.run(main())
```
36 changes: 36 additions & 0 deletions llmeter/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,39 @@
__version__ = version(__name__)
except PackageNotFoundError:
__version__ = "0.0.0"


def _register_s3_filesystem():
"""Register boto3 S3 backend if no other S3 backend is available."""
try:
import s3fs # noqa: F401

return # s3fs is installed, defer to it
except ImportError:
pass

try:
from obstore.fsspec import FsspecStore # noqa: F401

return # obstore is installed, defer to it
except ImportError:
pass

try:
import fsspec

fsspec.register_implementation(
"s3", "llmeter.s3fs.Boto3S3FileSystem", clobber=True
)
fsspec.register_implementation(
"s3a", "llmeter.s3fs.Boto3S3FileSystem", clobber=True
)
except Exception:
import logging

logging.getLogger(__name__).warning(
"Failed to register llmeter S3 filesystem backend", exc_info=True
)


_register_s3_filesystem()
Loading