Examples and tutorials

🍻 Examples and Tutorials

To easily understand and get your hands dirty with BEIR, we invite you to try our tutorials out 🚀 🚀

🍻 Google Colab

Name	Link
How to evaluate pre-trained models on BEIR datasets

🍻 Lexical Retrieval (Evaluation)

I highly recommend looking into Pyserini for reproducible & reliable BM25 implementations.

Name	Link	Upto Date
Pyserini 2CR for BEIR 🆕	Homepage	✅
BM25 Retrieval with Elasticsearch (Old)	evaluate_bm25.py	❌
Anserini-BM25 (Pyserini) Retrieval with Docker (Old)	evaluate_anserini_bm25.py	❌
Multilingual BM25 Retrieval with Elasticsearch (Old)	evaluate_multilingual_bm25.py	❌

🍻 Dense Retrieval (APIs, e.g. Cohere)

Name	Link	Upto Date
Exact-search retrieval using Cohere Embed v4.0 🆕	evaluate_cohere.py	✅
Exact-search retrieval using VoyageAI 🆕	evaluate_voyage.py	✅

🍻 Dense Retrieval (Evaluation)

Name	Link	Upto Date
Exact-search retrieval using any LoRA LLM retriever & VLLM 🆕	evaluate_lora_vllm.py	✅
Exact-search retrieval by saving embeddings and searching 🆕	evaluate_huggingface_pkl_embs.py	✅
Exact-search retrieval using HuggingFace 🆕	evaluate_huggingface.py	✅
Exact-search retrieval using LLM2Vec 🆕	evaluate_llm2vec.py	✅
Exact-search retrieval using NV-Embed (v2) 🆕	evaluate_nvembed.py	✅
Exact-search retrieval using any Sentence-BERT model 🆕	evaluate_sbert.py	✅
Exact-search retrieval using (dense) ANCE (Old)	evaluate_ance.py	❌
Exact-search retrieval using (dense) DPR (Old)	evaluate_dpr.py	❌
ANN and Exact-search using Faiss (Old)	evaluate_faiss_dense.py	❌
Retrieval using Binary Passage Retriver (BPR) (Old)	evaluate_bpr.py	❌
Dimension Reduction using PCA (Old)	evaluate_dim_reduction.py	❌

🍻 Sparse Retrieval (Evaluation)

Name	Link	Upto Date
Hybrid sparse retrieval using SPARTA (Old)	evaluate_sparta.py	❌
Sparse retrieval using docT5query and Pyserini (Old)	evaluate_anserini_docT5query.py	❌
Sparse retrieval using docT5query (MultiGPU) and Pyserini (Old)	evaluate_anserini_docT5query_parallel.py	❌
Sparse retrieval using DeepCT and Pyserini (outdated)	evaluate_deepct.py	❌

🍻 Reranking (Evaluation) --- Outdated

I highly recommend for reranking models, please look into Tevatron or Sentence-Transformers.

Name	Link	Upto Date
Reranking top-100 BM25 results with SBERT CE	evaluate_bm25_ce_reranking.py	❌
Reranking top-100 BM25 results with Dense Retriever	evaluate_bm25_sbert_reranking.py	❌

🍻 Dense Retrieval (Training) --- Outdated

I highly recommend for training state-of-the-art retrieval models, please look into Tevatron or Sentence-Transformers.

Name	Link	Upto Date
Train SBERT with Inbatch negatives	train_sbert.py	❌
Train SBERT with BM25 hard negatives	train_sbert_BM25_hardnegs.py	❌
Train MSMARCO SBERT with BM25 Negatives	train_msmarco_v2.py	❌
Train (SOTA) MSMARCO SBERT with Mined Hard Negatives	train_msmarco_v3.py	❌
Train (SOTA) MSMARCO BPR with Mined Hard Negatives	train_msmarco_v3_bpr.py	❌
Train (SOTA) MSMARCO SBERT with Mined Hard Negatives (Margin-MSE)	train_msmarco_v3_margin_MSE.py	❌

🍻 Question Generation --- Outdated

I highly recommend using state-of-the-art LLMs for question generation these days using vLLM.

Name	Link	Upto Date
Synthetic Query Generation using T5-model	query_gen.py	❌
(GenQ) Synthetic QG using T5-model + fine-tuning SBERT	query_gen_and_train.py	❌
Synthetic Query Generation using Multiple GPU and T5	query_gen_multi_gpu.py	❌

🍻 Benchmarking (Evaluation) --- Outdated

Name	Link	Upto Date
Benchmark BM25 (Inference speed)	benchmark_bm25.py	❌
Benchmark Cross-Encoder Reranking (Inference speed)	benchmark_bm25_ce_reranking.py	❌
Benchmark Dense Retriever (Inference speed)	benchmark_sbert.py	❌