A systematic study of parallel computing techniques applied to evaluating multi-hop Retrieval-Augmented Generation (RAG) pipelines on the HotpotQA benchmark — comparing sequential evaluation, naive process-pool parallelism, and a custom difficulty-aware dynamic scheduler, with a real LLM (Flan-T5), GPU-batched embeddings, and statistical validation.
A collaborative research project completed as coursework at FAST-NUCES, Islamabad.
Evaluating a RAG pipeline over a large QA benchmark is expensive: every example needs retrieval, generation, and one or more scoring steps. A single sequential pass over just 200 examples with a real LLM generator took 6.8 seconds per 50 examples — scaled to a full 7,405-example validation split, that's roughly 17 minutes per evaluation run, repeated dozens of times per day during active research. This project asks: can parallel and difficulty-aware scheduling meaningfully cut that cost without changing evaluation correctness?
Three evaluation scheduling strategies, benchmarked against each other on identical data:
- Sequential baseline — standard single-core evaluation loop
- Process-pool parallel —
multiprocessing.Poolwith naive round-robin task distribution - Dynamic difficulty-aware scheduler — sorts tasks by estimated per-example complexity (Longest-Job-First heuristic) before dispatching to the process pool, to avoid idle workers waiting on a late-arriving expensive task
On top of the scheduling comparison, the project adds:
- A real LLM generator (
google/flan-t5-base) replacing the mock baseline, run both sequentially and GPU-batched - GPU-batched sentence embeddings (
all-MiniLM-L6-v2) vs. CPU-sequential, swept across batch sizes 8–128 - Statistical robustness testing — 5 repeated runs with mean ± std to confirm results aren't noise
- An extended scaling study (n = 50 to 500) to empirically locate the Amdahl's Law crossover point
- A baseline comparison against classical BM25 and TF-IDF retrieval
| Finding | Result |
|---|---|
| Dynamic scheduler vs. naive parallel pool | 23.6% latency reduction (statistically significant, 8.5σ over 5 runs) |
| Flan-T5 vs. mock generator accuracy | 15.5% → 30.0% exact-match (81% relative improvement) |
| GPU-batched vs. CPU-sequential embedding | 107.2× speedup at batch size 64 |
| Amdahl crossover point | Theoretical n ≈ 409; dynamic scheduler reaches near-unity speedup by n = 360 — beating the naive pool's crossover by 140+ examples |
| BM25 vs. mock baseline | 2× accuracy (24.0% vs. 12.0%) at only 9× latency cost |
Full methodology, all tables, and statistical analysis are in the paper (paper/research-paper.pdf).
Python · PyTorch (CUDA) · HuggingFace transformers & datasets · sentence-transformers ·
multiprocessing · rank_bm25 · scikit-learn · matplotlib · Google Colab (NVIDIA T4 GPU)
notebook/rag_evaluation_scheduling.ipynb— full implementation: all three scheduling strategies, Flan-T5 integration, GPU-batched embedding experiments, statistical robustness runs, extended scaling study, and BM25/TF-IDF baseline comparison. Executed with saved outputs from the original Colab run.paper/research-paper.pdf— the accompanying IEEE-format research paper with full methodology, results tables, and discussion.
git clone https://github.com/<your-username>/parallel-difficulty-aware-rag-scheduling.git
cd parallel-difficulty-aware-rag-schedulingOpen notebook/rag_evaluation_scheduling.ipynb in Google Colab (GPU runtime recommended — a
Tesla T4 or equivalent) and run cells top to bottom. The notebook installs its own dependencies
(datasets, transformers, sentence-transformers, rank_bm25, etc.) in the first cells.
- Test process-pool scaling on hardware with more than 2 cores to see how the dynamic scheduler's advantage changes at higher parallelism
- Extend the Flan-T5 experiments beyond n = 50 (limited by Colab session time in this run)
- Try a richer difficulty proxy (question length, supporting-fact count, or a learned cost model) instead of context paragraph count
- Integrate the difficulty-aware scheduler with the BM25/TF-IDF pipelines directly
MIT