Skip to content

README.md

Latest commit

 

History

History
323 lines (234 loc) · 8.14 KB

README.md

File metadata and controls

323 lines (234 loc) · 8.14 KB
Raw

Exarch Benchmark Suite

Comprehensive performance benchmarks for the exarch archive extraction library.

Performance Targets

Operation Target Notes
TAR extraction 500 MB/s Throughput on uncompressed TAR
ZIP extraction 300 MB/s Throughput on stored ZIP
Path validation < 1 us Per entry validation
Symlink validation < 5 us Per symlink entry
Format detection < 10 us Per file

Quick Start

# 1. Generate benchmark fixtures (required first time)
./benches/fixtures/generate_fixtures.sh

# 2. Run all benchmarks
./benches/run_all.sh

# 3. View HTML report
open target/criterion/report/index.html

Benchmark Types

Rust Criterion Benchmarks

Located in crates/exarch-core/benches/:

  • extraction.rs - Archive extraction throughput

    • TAR (uncompressed, gzip, bzip2, xz, zstd)
    • ZIP (stored, deflate)
    • 7z
    • Different sizes (1MB, 10MB, 100MB)
    • Different structures (many files, nested dirs)

  • creation.rs - Archive creation throughput

    • TAR and ZIP creation
    • Compression level comparison
    • File count scaling

  • validation.rs - Security validation performance

    • Path validation (< 1 us target)
    • Symlink validation
    • Hardlink validation
    • Compression ratio checks (zip bomb detection)
    • Entry validator orchestration

  • progress.rs - Progress callback overhead measurement

Comparison Benchmarks

  • compare_python.py - exarch vs native Python tarfile/zipfile
  • compare_node.js - exarch-rs vs tar-fs/adm-zip

Running Benchmarks

Full Benchmark Suite

./benches/run_all.sh

Quick Benchmarks (fewer iterations)

./benches/run_all.sh --quick

Rust Only

./benches/run_all.sh --rust-only

Comparison Only

./benches/run_all.sh --compare

Individual Benchmark

# Run from project root
cd crates/exarch-core

# Run specific benchmark file
cargo bench --bench extraction

# Run specific benchmark group
cargo bench --bench validation -- path_validation

# Run with baseline comparison
cargo bench -- --save-baseline main
cargo bench -- --baseline main

Benchmark Fixtures

Generated by ./fixtures/generate_fixtures.sh:

Fixture Description Size
small_files.* 1000 files x 1KB ~1 MB
medium_files.* 100 files x 100KB ~10 MB
large_file.* 1 file x 100MB 100 MB
compressible_large.* Highly compressible 100MB ~100 MB
nested_dirs.* 20 levels deep, 3 files/level ~60 KB
many_files.* 10,000 tiny files ~200 KB
mixed.* Mixed file sizes ~10.5 MB

Formats generated: .tar, .tar.gz, .tar.bz2, .tar.xz, .tar.zst, .zip, .7z

Regenerating Fixtures

rm -rf benches/fixtures/*.tar* benches/fixtures/*.zip benches/fixtures/*.7z
./benches/fixtures/generate_fixtures.sh

Interpreting Results

Criterion Reports

Criterion generates detailed HTML reports in target/criterion/:

  • Throughput - MB/s or ops/s
  • Mean/Median - Average execution time
  • Std Dev - Variability
  • Outliers - Unusual measurements
  • Change - Regression/improvement vs baseline

Performance Targets

Check the benchmark output against targets:

path_validation/simple:
  time: 800 ns (target: < 1000 ns) [PASS]

tar_extraction/large_100mb:
  throughput: 520 MB/s (target: 500 MB/s) [PASS]

Regression Detection

# Save baseline
cargo bench -- --save-baseline before-change

# Make changes, then compare
cargo bench -- --baseline before-change

# Look for "Performance has regressed" warnings

CI Integration

Add to CI workflow:

- name: Run benchmarks
  run: |
    ./benches/fixtures/generate_fixtures.sh
    cargo bench -- --noplot

- name: Check for regressions
  run: |
    # Fail if any benchmark regressed by >10%
    cargo bench -- --baseline main | grep -q "regressed" && exit 1 || exit 0

Adding New Benchmarks

  1. Add benchmark function in appropriate file:
fn benchmark_new_feature(c: &mut Criterion) {
    let mut group = c.benchmark_group("new_feature");
    group.throughput(Throughput::Bytes(size));

    group.bench_function("variant_a", |b| {
        b.iter(|| {
            // Code to benchmark
        });
    });

    group.finish();
}
  1. Add to criterion_group:
criterion_group!(
    benches,
    // ... existing benchmarks
    benchmark_new_feature,
);
  1. If new fixture needed, update fixtures/generate_fixtures.sh

Troubleshooting

Fixtures not found

./benches/fixtures/generate_fixtures.sh

Python benchmarks skipped

cd crates/exarch-python
maturin develop --release

Node.js benchmarks skipped

cd crates/exarch-node
npm run build
npm install tar adm-zip  # For comparison

High variance in results

  • Close other applications
  • Run multiple times
  • Use --warm-up-time 5 for longer warmup
  • Consider --measurement-time 10 for more samples

Criterion version mismatch

Ensure workspace uses consistent criterion version:

# Cargo.toml
[workspace.dependencies]
criterion = "0.5"

Files

benches/
├── README.md                  # This file
├── run_all.sh                # Main benchmark runner
├── compare_python.py         # Python comparison script
├── compare_node.js           # Node.js comparison script
├── fixtures/
│   ├── generate_fixtures.sh  # Fixture generator
│   └── *.tar, *.zip, *.7z    # Generated test archives
└── BENCHMARK_RESULTS.md      # Generated results (after run)

crates/exarch-core/benches/
├── extraction.rs             # Extraction benchmarks
├── creation.rs               # Creation benchmarks
├── validation.rs             # Validation benchmarks
└── progress.rs               # Progress callback benchmarks

Benchmark Results (v0.2.1)

Python Comparison (exarch vs tarfile/zipfile)

Archive exarch (ms) Native (ms) Speedup Throughput
TAR small (1MB, 1000 files) 114.6 160.9 1.40x 9 MB/s
TAR+GZIP small 115.3 164.4 1.43x 9 MB/s
TAR medium (10MB) 15.9 19.1 1.20x 614 MB/s
TAR+GZIP medium 19.1 24.7 1.29x 512 MB/s
TAR large (100MB) 20.5 23.3 1.14x 4,885 MB/s
TAR+GZIP large 39.5 36.6 0.93x 2,531 MB/s
ZIP small (1MB) 117.6 110.4 0.94x 8 MB/s
ZIP medium (10MB) 18.5 14.7 0.80x 527 MB/s
ZIP large (100MB) 42.4 31.5 0.74x 2,361 MB/s

Average speedup: 1.10x faster than Python tarfile/zipfile

Node.js Comparison (exarch-rs vs tar/adm-zip)

Archive exarch (ms) Native (ms) Speedup Throughput
TAR small (1MB, 1000 files) 112.0 102.5 0.92x 9 MB/s
TAR+GZIP small 111.7 90.9 0.81x 9 MB/s
TAR medium (10MB) 16.4 10.8 0.66x 594 MB/s
TAR+GZIP medium 22.6 27.0 1.20x 433 MB/s
TAR large (100MB) 32.3 21.5 0.67x 3,100 MB/s
TAR+GZIP large 45.4 122.6 2.70x 2,205 MB/s
ZIP small (1MB) 113.4 136.1 1.20x 9 MB/s
ZIP medium (10MB) 17.5 51.3 2.93x 557 MB/s
ZIP large (100MB) 76.4 358.4 4.69x 1,309 MB/s

Average speedup: 1.75x faster than Node.js tar/adm-zip

Performance vs Targets

Metric Target Achieved Status
TAR extraction 500 MB/s 2,136 MB/s ✅ 4x target
ZIP extraction 300 MB/s 1,444 MB/s ✅ 5x target
Path validation < 1 µs ~85 ns ✅ 12x better

Optimizations (v0.2.1)

  • Directory caching — Reduces mkdir syscalls by ~95% via FxHashSet caching
  • Atomic permission setting — Sets Unix permissions during file creation (1 syscall vs 2)

Note

Results measured on Apple M1 Pro (macOS 25.1, Rust 1.92). Performance varies by hardware.

Related Documentation