Skip to content

benchmark-results.md

Latest commit

 

History

History
154 lines (108 loc) · 6.04 KB

benchmark-results.md

File metadata and controls

154 lines (108 loc) · 6.04 KB
Raw

VectorFlow SDK — Benchmark Results

Test Setup

Methodology: Simulated ingestion with realistic Azure service latencies:

Parameter Value Rationale
Embedding API latency 150ms per batch Real Azure OpenAI p50 latency
Cosmos DB write latency 10ms per record Real single-document upsert p50
RU per vector upsert 30 RU Typical for 1536-dim vector document
Embedding dimensions 1536 text-embedding-ada-002
VectorFlow RU budget 400 RU/s Conservative limit
VectorFlow batch size Up to 100 texts/API call Azure OpenAI batch limit

Naive approach: Embed one chunk at a time → write one record at a time → sequential, no batching.

VectorFlow SDK: Batch embeddings (up to 100/call) → buffer in memory → adaptive micro-batch writes.

Results

Summary Table

Chunks Doc Size Naive Duration VectorFlow Duration Speedup API Calls (Naive → VF) Write Batches (Naive → VF)
50 ~20K tokens 8.77s 5.07s 1.7x 50 → 1 (98% fewer) 50 → 8
200 ~80K tokens 34.96s 18.89s 1.9x 200 → 2 (99% fewer) 200 → 33
500 ~200K tokens 86.65s 46.95s 1.8x 500 → 5 (99% fewer) 500 → 83
1000 ~400K tokens ~173s* ~94s* ~1.8x 1000 → 10 (99% fewer) 1000 → ~165

*1000-chunk values projected from observed linear scaling.

Detailed Breakdown

50 Chunks (~20K tokens, small document)

Metric Naive VectorFlow Improvement
Total duration 8.77s 5.07s 1.7x faster
Embedding API calls 50 1 98% fewer
Total RU consumed 1,500 1,440 ~4% less
Avg latency/chunk 175.5ms 101.4ms 42% lower
Write batches 50 8 84% fewer
Peak RU/s 171 284 Controlled burst

200 Chunks (~80K tokens, medium document)

Metric Naive VectorFlow Improvement
Total duration 34.96s 18.89s 1.9x faster
Embedding API calls 200 2 99% fewer
Total RU consumed 6,000 5,940 ~1% less
Avg latency/chunk 174.8ms 94.4ms 46% lower
Write batches 200 33 84% fewer
Peak RU/s 172 314 Controlled burst

500 Chunks (~200K tokens, large document)

Metric Naive VectorFlow Improvement
Total duration 86.65s 46.95s 1.8x faster
Embedding API calls 500 5 99% fewer
Total RU consumed 15,000 14,940 ~0.4% less
Avg latency/chunk 173.3ms 93.9ms 46% lower
Write batches 500 83 83% fewer
Peak RU/s 173 318 Controlled burst

Key Findings

1. Embedding API Call Reduction: 98-99%

The most dramatic improvement. VectorFlow batches up to 100 texts per API call:

Naive:      500 chunks → 500 API calls × 150ms = 75s of embedding time
VectorFlow: 500 chunks → 5 API calls × 150ms  = 0.75s of embedding time

Cost impact: At $0.0001/1K tokens (ada-002 pricing), the cost per API call is the same — but the latency saved is massive because network round-trips are amortized.

2. Write Batch Consolidation: 83-84% fewer operations

Instead of 500 individual Cosmos upserts, VectorFlow consolidates into ~83 coordinated micro-batches. This:

  • Reduces network round-trips
  • Allows Cosmos to update its vector index more efficiently
  • Keeps RU consumption smooth rather than spiky

3. Throughput: ~1.8x overall speedup

VectorFlow achieves consistent ~1.8x throughput improvement across all document sizes. The speedup comes from:

  • Parallel embedding batches (3 concurrent calls)
  • Amortized write scheduling
  • Pipeline overlap (embedding while writing previous batch)

4. RU Consumption: Same total, better distribution

Total RUs are nearly identical (same number of records written) — but the distribution is different:

Pattern Naive VectorFlow
Burst behavior Constant low (171 RU/s) Controlled peaks (284-318 RU/s) within budget
Impact on reads Spread over longer time, blocks reads longer Shorter window, scheduler respects budget
Index update pattern 500 individual index updates 83 batched index updates (more efficient)

The scheduler keeps peak RU/s within the configured 400 RU/s budget, ensuring read queries aren't starved.

5. With Redis Cache (projected)

For re-ingestion scenarios (document re-upload, retry after failure):

Scenario Without Cache With Cache Savings
Re-ingest same 500 chunks 5 API calls, ~0.75s 0 API calls, ~0ms 100% embedding cost eliminated
Partial overlap (50% new) 5 API calls 3 API calls 40% fewer calls

When VectorFlow Shines Most

Scenario Benefit
Large documents (100+ chunks) 99% fewer API calls, 1.8x faster
Concurrent users uploading Scheduler prevents RU spikes from impacting searches
Re-ingestion/retry Redis cache eliminates redundant OpenAI costs
Serverless Cosmos DB RU budget control prevents throttling (429s)
Batch imports (migration) Controlled throughput without burning provisioned RUs

How to Run

cd tests/VectorFlow.Benchmarks
dotnet run

The benchmark uses simulated services (no Azure credentials required) with realistic latencies to produce reproducible results.

Limitations of This Benchmark

  1. Simulated services — Real Azure latency varies (p50 vs p99, cold starts, throttling)
  2. No contention — Benchmark runs in isolation; real benefit of RU scheduling shows under concurrent read load
  3. No Redis test — Cache benefit is projected, not measured
  4. Linear scaling assumed — Real Cosmos DB behavior at scale may show non-linear RU patterns

For production validation, run the benchmark against real Azure services with the --live flag (not yet implemented).