Multi-Platform Benchmark Sandbox Infrastructure

[fajarhide]

Feature Idea

Build a comprehensive, reproducible benchmark sandbox that enables OMNI and its competitors (RTK, lean-ctx, caveman, headroom) to be tested across multiple platforms (Linux x86/ARM, macOS, Kubernetes) with immutable, versioned corpora.

Problem Statement

Current limitations of OMNI's existing benchmarking:

1. Single workload: Only one developer's real usage (70 sessions, 9,478 traces)
2. Non-reproducible: execution_traces prunes at 7 days; old benchmarks can't be re-derived
3. Platform bias: Measurements only on one machine (maintainer's dev environment)
4. Manual process: No automation or regression detection
5. Competitor testing: RTK, lean-ctx, caveman, headroom tested ad-hoc with different setups
6. No CI/CD integration: Can't run benchmarks on every PR to detect regressions

This blocks confident decision-making about:

• Whether a change actually improved performance
• How we compare across platforms (Linux vs macOS)
• Whether future improvements are real or corpus-dependent (Pin a benchmark corpus: every published figure was measured on traces that no longer exist #704 proved this matters)

Proposed Solution

Create omni-sandbox: A standalone Rust crate and orchestration framework that:

1. Manages versioned, immutable corpora (stored in S3, hashed for reproducibility)
2. Runs benchmarks locally (using existing run_inner() as library)
3. Containerizes competitors (Docker, cross-compile to Linux ARM/x86)
4. Schedules CI jobs (GitHub Actions matrix on Ubuntu + macOS)
5. Aggregates results (head-to-head tables, regression detection, HTML reports)
6. Publishes results (S3, GitHub Pages, PR comments)

Architecture

┌──────────────────────────────────────────────────────────┐
│        OMNI Benchmark Orchestrator (omni-sandbox)       │
│  - Corpus registry (versioned, immutable, OCI-friendly)  │
│  - Job scheduler (local, Docker, K8s, GitHub Actions)   │
│  - Result aggregator + comparison logic                 │
└──────────────────────────────────────────────────────────┘
         │
    ┌────┼────┬────────┬──────────┬──────┐
    ↓    ↓    ↓        ↓          ↓      ↓
  Local Docker K8s  GitHub-   EC2/  Fallback
  (dev)  (CI)  (prod) Actions Cloud  (pipe)

Implementation Scope

Phase 1: Local Sandbox (Weeks 1-2, ~40 hours)

Goal: One developer can run OMNI vs competitors locally, per-platform

Deliverables:

• New crate: omni-sandbox/ with Cargo.toml, src/lib.rs
• src/runner.rs: Execute benchmarks (call OMNI directly via library, subprocess for competitors)
• src/corpus.rs: Load/cache versioned corpora
• src/comparator.rs: Compare results, generate markdown/HTML
• benches/ reuses existing bench_replay.rs logic
• CLI: cargo run -p omni-sandbox -- run --corpus latest --tools omni,rtk,lean-ctx
• Output: JSON results + markdown summary table

File structure:

omni-sandbox/
├── Cargo.toml
├── src/
│   ├── lib.rs
│   ├── orchestrator.rs        # Job scheduling
│   ├── corpus.rs              # Corpus management
│   ├── runner.rs              # Execute a single benchmark
│   ├── comparator.rs          # Compare results
│   └── platform.rs            # Platform detection
├── benches/
│   └── replay_fixture.rs      # Main benchmark (reuses existing)
├── fixtures/
│   ├── corpus-0b63218ef78a1edb.json.gz  # Immutable
│   └── corpus-registry.yaml
└── tests/
    └── integration_test.rs

Key code: src/runner.rs

pub struct BenchmarkRunner {
    tool: ToolConfig,           // omni, rtk, lean-ctx, etc
    corpus: CorpusHandle,       // Immutable payload
    platform: TargetPlatform,   // linux-x86, macos-arm64, etc
}

impl BenchmarkRunner {
    pub async fn execute(&self) -> BenchResult {
        match &self.platform {
            TargetPlatform::Local => self.run_subprocess().await,
            TargetPlatform::Docker(_) => self.run_docker().await,
        }
    }
    
    async fn run_omni_direct(&self, traces: &[ExecutionTrace]) -> Result<ToolOutput> {
        // Reuse OMNI's own pipeline (no subprocess overhead)
        use omni::hooks::pipe::run_inner;
        use omni::store::sqlite::Store;
        
        let store = Arc::new(Store::open_path(work_dir.join("bench.db"))?);
        let mut total_input = 0;
        let mut total_output = 0;
        
        for trace in traces {
            let mut output = Vec::new();
            total_input += trace.raw.len();
            run_inner(trace.raw.as_bytes(), &mut output, ...)?;
            total_output += output.len();
        }
        
        Ok(ToolOutput { total_input, total_output, ... })
    }
}

Corpus registry: fixtures/corpus-registry.yaml

version: "1.0"

corpora:
  "0b63218ef78a1edb":
    name: "Aug 24, 2026 - OMNI dev session"
    date: 2026-08-24
    traces: 9478
    bytes: 8458937
    tags: [shell, claude_code, real-usage]
    url: "s3://omni-benchmarks/corpus-0b63218ef78a1edb.json.gz"
    sha256: "abc123..."

Test: Can run locally

cargo run -p omni-sandbox -- run \
  --corpus 0b63218ef78a1edb \
  --tools omni,rtk \
  --output /tmp/results.json

# Output:
# ✅ Loaded corpus: 9,478 traces (8.5 MB)
# ✅ omni: 5.1% reduction (8.49 MB → 8.07 MB)
# ✅ rtk: 2.1% reduction
# Report written to /tmp/results.json

Phase 2: Docker Multi-Platform (Weeks 3-4, ~40 hours)

Goal: Build and test on Linux x86, Linux ARM, macOS (for CI)

Deliverables:

• docker/Dockerfile.base (Rust + toolchains)
• docker/Dockerfile.rtk (compile RTK from source or download binary)
• docker/Dockerfile.lean-ctx
• docker/docker-compose.bench.yml (orchestrate runs on multiple platforms)
• docker buildx setup for cross-compile (linux/amd64, linux/arm64, darwin/arm64)
• GitHub Actions: .github/workflows/bench-docker.yml
• Result aggregation + report generation

Docker Compose

version: '3.8'

services:
  bench-linux-x86:
    build:
      context: .
      dockerfile: Dockerfile.base
      args:
        RUST_TARGETS: "x86_64-unknown-linux-gnu"
    image: omni-bench:linux-x86
    environment:
      - OMNI_BENCH_CORPUS=0b63218ef78a1edb
      - OMNI_BENCH_TOOLS=omni,rtk,lean-ctx
    volumes:
      - ./results:/results
      - ./fixtures:/fixtures:ro
    command: omni-sandbox run --corpus 0b63218ef78a1edb --output /results/linux-x86.json

  bench-linux-arm64:
    build:
      context: .
      dockerfile: Dockerfile.base
      args:
        RUST_TARGETS: "aarch64-unknown-linux-gnu"
    image: omni-bench:linux-arm64
    volumes:
      - ./results:/results
    command: omni-sandbox run --output /results/linux-arm64.json

  report:
    image: omni-bench:linux-x86
    depends_on:
      - bench-linux-x86
      - bench-linux-arm64
    volumes:
      - ./results:/results
    command: omni-sandbox report --input /results/*.json --output /results/report.html

GitHub Actions: .github/workflows/bench-docker.yml

name: Docker Multi-Platform Benchmark

on:
  workflow_dispatch:
  schedule:
    - cron: '0 2 * * 0'  # Weekly on Sunday

jobs:
  build-and-bench:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        platform: [linux/amd64, linux/arm64]
    
    steps:
      - uses: actions/checkout@v4
      
      - uses: docker/setup-buildx-action@v2
      
      - name: Build benchmark image
        uses: docker/build-push-action@v4
        with:
          context: .
          file: ./docker/Dockerfile.base
          platforms: ${{ matrix.platform }}
          push: false
          load: true
          tags: omni-bench:latest
      
      - name: Run benchmark
        run: docker run -v /tmp/results:/results omni-bench:latest \
          omni-sandbox run --output /results/${{ matrix.platform }}.json
      
      - name: Upload results
        uses: actions/upload-artifact@v3
        with:
          name: bench-results
          path: /tmp/results/*.json

Phase 3: GitHub Actions Multi-Platform (Weeks 5-6, ~50 hours)

Goal: Run benchmarks on every PR and nightly, detect regressions, post results

Deliverables:

• .github/workflows/bench-pr.yml (quick smoke test on PR)
• .github/workflows/bench-nightly.yml (full suite, all platforms, all tools)
• Regression detection (compare against baseline, fail if >5% worse)
• PR comments with summary (embed table + link to full report)
• S3 uploads (benchmarks/v0.8.0/results.json)

Key workflow: .github/workflows/bench-nightly.yml

name: Nightly Comprehensive Benchmark

on:
  schedule:
    - cron: '0 2 * * *'
  workflow_dispatch:

jobs:
  benchmark:
    strategy:
      matrix:
        include:
          - platform: ubuntu-latest
            target: x86_64-unknown-linux-gnu
            name: linux-x86
          - platform: macos-latest
            target: x86_64-apple-darwin
            name: macos-x86
          - platform: macos-14
            target: aarch64-apple-darwin
            name: macos-arm64
    
    runs-on: ${{ matrix.platform }}
    
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain@stable
        with:
          targets: ${{ matrix.target }}
      
      - name: Cache cargo registry
        uses: actions/cache@v3
        with:
          path: ~/.cargo/registry
          key: ${{ runner.os }}-cargo-registry
      
      - name: Install competitor tools
        run: |
          # RTK
          cargo install rtk --version 0.45.0
          # lean-ctx
          cargo install lean-ctx --version 3.9.18
          # caveman (download binary)
          curl -L https://github.com/.../caveman/releases/download/v1.0.0/caveman-${{ matrix.name }} -o /tmp/caveman
          chmod +x /tmp/caveman
      
      - name: Run benchmarks
        run: cargo bench -p omni-sandbox --release \
          -- --target ${{ matrix.target }} \
          --output /tmp/bench-${{ matrix.name }}.json
      
      - name: Upload results
        uses: actions/upload-artifact@v3
        with:
          name: bench-${{ matrix.name }}
          path: /tmp/bench-*.json
  
  aggregate:
    needs: benchmark
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/download-artifact@v3
        with:
          name: bench-*
          path: ./results/
      
      - name: Generate comparison report
        run: |
          cargo run -p omni-sandbox -- report \
            --input results/*.json \
            --corpus 0b63218ef78a1edb \
            --output report.html \
            --baseline s3://omni-benchmarks/latest/results.json
      
      - name: Detect regressions
        run: |
          # Fail if any platform shows >5% regression
          python3 scripts/check_regression.py report.html 0.05
      
      - name: Upload to S3
        run: |
          aws s3 cp report.html s3://omni-benchmarks/$(date +%Y-%m-%d)/report.html
          aws s3 cp results/*.json s3://omni-benchmarks/$(date +%Y-%m-%d)/
      
      - name: Update GitHub Pages
        run: |
          # Publish to https://fajarhide.github.io/omni-benchmarks/
          cp report.html docs/benchmarks/latest.html
          git config user.name "benchmark-bot"
          git add docs/benchmarks/latest.html
          git commit -m "chore: update benchmark results"
          git push

PR workflow: .github/workflows/bench-pr.yml

name: Quick Benchmark (PR)

on:
  pull_request:
    paths:
      - 'src/hooks/**'
      - 'src/distillers/**'
      - 'src/ledger/**'

jobs:
  quick-bench:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain@stable
      
      - name: Run quick benchmark (baseline + head)
        run: |
          # Baseline (merge-base)
          git checkout $(git merge-base origin/main HEAD)
          cargo bench -p omni-sandbox --release -- \
            --corpus small --output /tmp/baseline.json
          
          # Head (PR)
          git checkout -
          cargo bench -p omni-sandbox --release -- \
            --corpus small --output /tmp/pr.json
      
      - name: Comment with results
        uses: actions/github-script@v6
        with:
          script: |
            const fs = require('fs');
            const baseline = JSON.parse(fs.readFileSync('/tmp/baseline.json'));
            const pr = JSON.parse(fs.readFileSync('/tmp/pr.json'));
            
            const delta = (pr.savings - baseline.savings) / baseline.savings * 100;
            const status = delta > 0 ? '✅' : '❌';
            
            github.rest.issues.createComment({
              ...context.repo,
              issue_number: context.issue.number,
              body: `${status} **Benchmark**: ${pr.savings}% (${delta > 0 ? '+' : ''}${delta.toFixed(1)}%)\n\n[View full report](${pr.report_url})`
            });

Phase 4: Corpus Management & Registry (Weeks 6-8, ~50 hours)

Goal: Versioned, immutable, reproducible corpora with generator support

Deliverables:

• src/corpus.rs: Registry loading, S3 download/cache, SHA256 validation
• benches/fixtures/corpus-registry.yaml (YAML manifest of all corpora)
• Corpus generators for synthetic workloads (cargo builds, git operations)
• Upload infrastructure (S3 bucket + CI job to publish)
• Documentation: How to add new corpus, how benchmarks are reproducible

Corpus registry: benches/fixtures/corpus-registry.yaml

version: "1.0"

corpora:
  "0b63218ef78a1edb":
    name: "Aug 24, 2026 - OMNI dev session (real usage)"
    description: "9,478 traces from actual development work"
    date: 2026-08-24
    traces: 9478
    bytes: 8458937
    tags: [shell, claude_code, real-usage, v0.7.8]
    
    files:
      - name: "corpus-0b63218ef78a1edb.json.gz"
        size: 2100000
        sha256: "abc123def456..."
        url: "s3://omni-benchmarks/corpora/0b63218ef78a1edb.json.gz"
        
    experiments:
      - name: "v0.7.8 baseline (Aug 26, 2026)"
        date: 2026-08-26
        results: s3://omni-benchmarks/v0.7.8-0b63218ef78a1edb.json
        
      - name: "v0.8.0 candidate (Sep 1, 2026)"
        date: 2026-09-01
        results: s3://omni-benchmarks/v0.8.0-0b63218ef78a1edb.json

  "synthetic-cargo-builds":
    name: "Synthetic: 1000 random cargo builds"
    description: "Generated corpus with realistic build output"
    date: 2026-08-25
    traces: 1000
    bytes: 50000000
    tags: [synthetic, build-only]
    
    generator:
      script: "benches/generators/cargo_build.rs"
      seed: 42
      variations: 10  # 10 different build scenarios
    
    files:
      - name: "synthetic-cargo-builds.json.gz"
        sha256: "xyz789..."
        url: "s3://omni-benchmarks/corpora/synthetic-cargo-builds.json.gz"

Loading corpora: src/corpus.rs

pub struct CorpusRegistry {
    registry: serde_yaml::Value,
}

impl CorpusRegistry {
    pub async fn load_corpus(&self, id: &str) -> Result<Vec<ExecutionTrace>> {
        let meta = &self.registry["corpora"][id];
        let url = meta["files"][0]["url"].as_str().ok_or("no url")?;
        let expected_sha = meta["files"][0]["sha256"].as_str()?;
        
        // Download or use cache
        let path = self.cache_dir().join(id).with_extension("json.gz");
        if !path.exists() {
            eprintln!("Downloading corpus {}...", id);
            download_from_s3(url, &path).await?;
        }
        
        // Verify SHA256
        let actual_sha = sha256::digest_file(&path)?;
        if actual_sha != expected_sha {
            bail!("Corpus {} corrupted (SHA mismatch)", id);
        }
        
        // Decompress & parse
        let file = std::fs::File::open(&path)?;
        let gz = flate2::read::GzDecoder::new(file);
        let traces: Vec<ExecutionTrace> = serde_json::from_reader(gz)?;
        
        eprintln!("✅ Loaded {} traces ({} bytes)", traces.len(), id);
        Ok(traces)
    }
}

Testing Strategy

#[cfg(test)]
mod bench_tests {
    use super::*;
    
    /// Phase 1: Local execution works
    #[tokio::test]
    async fn runner_executes_omni_directly() {
        let runner = BenchmarkRunner::new_omni("0b63218ef78a1edb");
        let result = runner.execute().await.unwrap();
        
        assert!(result.savings_pct > 0.0);
        assert_eq!(result.tool_name, "omni");
    }
    
    /// Corpus loads correctly and is validated
    #[tokio::test]
    async fn corpus_loads_and_validates() {
        let registry = CorpusRegistry::load().await.unwrap();
        let corpus = registry.load_corpus("0b63218ef78a1edb").await.unwrap();
        
        assert_eq!(corpus.len(), 9478);
        assert!(corpus[0].raw_output.len() > 0);
    }
    
    /// Comparison generates valid tables
    #[test]
    fn comparator_generates_markdown_table() {
        let results = vec![
            BenchResult { tool_name: "omni".into(), savings_pct: 5.1, ... },
            BenchResult { tool_name: "rtk".into(), savings_pct: 2.1, ... },
        ];
        
        let table = Comparator::new(results).to_markdown();
        assert!(table.contains("omni"));
        assert!(table.contains("rtk"));
        assert!(table.contains("5.1%"));
    }
    
    /// Regression detection works
    #[test]
    fn regression_detector_catches_degradation() {
        let baseline = 5.1;  // Last release
        let current = 4.8;   // Current PR
        
        let regression = check_regression(baseline, current, 0.05);
        assert!(regression.is_some());  // >5% is failure
    }
}

Documentation Updates Needed

• docs/website/src/develop/benchmarking.md (user guide to run local benchmarks)
• omni-sandbox/README.md (architecture, setup, examples)
• CONTRIBUTING.md section on benchmark-driven development
• S3 bucket setup guide (for maintainers)
• GitHub Pages integration (publish results)

Related Issues

• Pin a benchmark corpus: every published figure was measured on traces that no longer exist #704: Freeze corpus so runs are comparable (this issue fully solves that)
• Contribute OMNI upstream to hermes-agent and openclaw, and write the pitch that earns the slot #629: Contribute OMNI upstream (benchmarks prove performance before contribution)
• sst/opencode is the highest-capability host surveyed and has no token plugin: tool.execute.after plus a whole-message-array hook #630: OpenCode integration (need benchmarks to validate cross-platform compatibility)
• Prove OMNI is cache-safe: 98.8% hit rate over 54,421 requests, and we have never said it #610: Prove cache-safety claim (add benchmark for this specific metric)

Acceptance Criteria

Phase 1 (Local)

• omni-sandbox crate compiles without external dependencies (except Rust std + Cargo)
• runner.rs calls OMNI via omni::hooks::pipe::run_inner() (no subprocess overhead)
• Can load corpus from local file, benchmark, output JSON
• Can compare 2-3 tools (omni, rtk) and generate markdown table
• cargo test -p omni-sandbox passes with realistic corpus

Phase 2 (Docker)

• docker-compose.bench.yml runs on all supported platforms (x86, ARM)
• Cross-compiled binaries work correctly
• Results aggregated into single HTML report
• GitHub Actions workflow runs without secrets (S3 upload optional)

Phase 3 (CI/CD)

• PR workflow runs in <5 min (quick corpus only)
• Nightly workflow runs in <30 min (full corpus, all platforms)
• Regression detector blocks merges if >5% worse
• Results published to S3 + GitHub Pages

Phase 4 (Corpus)

• Corpus registry YAML loads successfully
• Corpora validated via SHA256 on download
• At least 3 different corpora available (real usage + 2 synthetic)
• Can generate new corpus programmatically

Cross-Cutting

• Zero data loss (identical input → identical output every run)
• Reproducible across platforms (same corpus → same baseline)
• Regression detected if savings drops >5% p99
• All benchmarks documented in this file or linked from here

Effort Estimate

• Phase 1 (Local): ~40 hours (1 week, 1 dev)
• Phase 2 (Docker): ~40 hours (1 week, 1 dev)
• Phase 3 (CI/CD): ~50 hours (1.5 weeks, 1 dev + DevOps)
• Phase 4 (Corpus): ~50 hours (1.5 weeks, 1 dev)
• Total: ~180 hours (6 weeks, 1 FTE)

Context

This solves a critical gap: we cannot confidently claim performance improvements without reproducible, multi-platform benchmarks. Every release currently publishes figures that depend on:

1. One developer's specific workload (corpus)
2. One machine's configuration (platform)
3. Manual testing (no regression detection)

This issue turns benchmarking from art into science, and unlocks:

• Confidence: Changes are measured, not guessed
• Transparency: Competitors tested fairly (same corpus, same build, same env)
• Reproducibility: Any developer can verify any claim
• Upstream contributions: We can prove OMNI's value before contributing to vLLM/Ollama/etc

Related Discussion

Users from #629 (upstream contributions) requested "objective benchmarks to show project value". This issue delivers exactly that infrastructure.

[fajarhide]

Triaged priority: low, stage: later. Most of Phase 1 already exists, and the phase
that would matter is blocked on something this does not provide.

What is already here: scripts/bench.sh runs our arm plus rtk, lean-ctx, headroom
and caveman over a frozen corpus, skipping any arm whose binary is absent rather than
failing, and writes docs/benchmarks/<version>.json (0.7.7 and 0.7.8 are committed).
The corpus is bench-corpus/ at 18 MB, 9,478 traces with a manifest carrying the byte
buckets and per-class bytes. So "build a runner, load a versioned corpus, compare three
tools, emit JSON" is not the gap.

The gap named in ROADMAP.md is a second corpus from a machine we do not own. Every
figure we can publish is n=1 on one developer's workload, and plan.md §11.5 refuses to
publish an n=1 number. Running the same 9,478 traces on ARM and macOS in CI measures
portability, not workload diversity, so it does not unblock P4 or P5. That is why PD
(distribution) sits ahead of them: the input is users.

The piece worth pulling out on its own is regression detection on the arm we already run,
against the committed baselines. That is a workflow and a threshold, not a new crate, an
S3 registry, or 180 hours.