AI SDK Benchmark Suite

Comprehensive benchmarking tool to compare binary size, latency, memory usage, and throughput between aisdk and rig Rust AI libraries.

Setup

1. Copy .env.example to .env and add your OpenAI API key:

   cp .env.example .env
   # Edit .env and set OPENAI_API_KEY

2. Build and run benchmarks with specific feature flags

Features

The benchmark suite supports the following feature flags for isolation:

AISDK Features

• aisdk-qa - Basic Q&A without streaming
• aisdk-qa-streaming - Streaming Q&A responses
• aisdk-embedding - Basic embedding generation
• aisdk-embedding-full - Full embedding with feature enabled

RIG Features

• rig-qa - Basic Q&A without streaming
• rig-qa-streaming - Streaming Q&A responses
• rig-embedding - Basic embedding generation
• rig-embedding-full - Full embedding with feature enabled

Running Benchmarks

Run a single benchmark:

cargo run --release --features aisdk-qa

Run multiple scenarios:

cargo run --release --features aisdk-qa,aisdk-qa-streaming,aisdk-embedding,aisdk-embedding-full

Compare libraries:

cargo run --release --features aisdk-qa,rig-qa

Full benchmark suite:

cargo run --release --all-features

Output

The benchmark produces two outputs:

1. Human-readable table printed to stdout showing:

   • Scenario name
   • Library (aisdk/rig)
   • Scenario type (Basic Q&A, Streaming, Embedding, etc.)
   • Latency (ms)
   • Memory peak (MB)
   • Throughput (ops/sec)
   • Binary size (MB)

2. JSON file (benchmark_results.json) for programmatic analysis

Metrics Explained

• Latency (ms): Total wall-clock time for the operation
• Memory (MB): Peak resident set size during execution
• Throughput (ops/sec): Operations per second (1/latency)
• Binary Size (MB): Release binary size on disk

Requirements

• Rust 1.70+
• OpenAI API key
• Linux/macOS/Windows with sysinfo support

Notes

• Each feature flag ensures only the specified library is compiled into the binary, allowing for accurate binary size comparisons
• Binary size includes all dependencies required by each library
• Memory measurements use platform-specific implementations
• Latency is measured using wall-clock time for the entire operation including network I/O