Skip to content

Commit fcbeffa

Browse files
deesatzedclaude
andcommitted
Update regression baselines with dense_only and latest benchmark scores
New baselines: - Qwen3-8B dense_only: SciFact 0.765 (-0.4% vs embed-only), NFCorpus 0.372 (-3.1%) - Qwen3-8B dense_dominant: SciFact 0.733 (-4.6%) - Qwen3-8B strong_hybrid: NFCorpus 0.344 (-10.4%) - MiniLM auto-calibrate: SciFact 0.741 - MiniLM bge-reranker-v2-m3: SciFact 0.737 dense_only confirmed as best strategy for strong embeddings on all corpus types. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent a74a938 commit fcbeffa

1 file changed

Lines changed: 42 additions & 0 deletions

File tree

‎benchmarks/mteb/baselines/regression_baselines.json‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,48 @@
6262
"metric": "ndcg_10",
6363
"value": 0.60207,
6464
"timestamp": "2026-04-30T00:00:00+00:00"
65+
},
66+
{
67+
"task": "NFCorpus",
68+
"config": "qwen3-8b-dense-only",
69+
"metric": "ndcg_10",
70+
"value": 0.37167,
71+
"timestamp": "2026-05-01T00:00:00+00:00"
72+
},
73+
{
74+
"task": "NFCorpus",
75+
"config": "qwen3-8b-strong-hybrid",
76+
"metric": "ndcg_10",
77+
"value": 0.34436,
78+
"timestamp": "2026-05-01T00:00:00+00:00"
79+
},
80+
{
81+
"task": "SciFact",
82+
"config": "qwen3-8b-dense-dominant",
83+
"metric": "ndcg_10",
84+
"value": 0.73328,
85+
"timestamp": "2026-05-01T00:00:00+00:00"
86+
},
87+
{
88+
"task": "SciFact",
89+
"config": "miniLM-hybrid-auto-calibrate",
90+
"metric": "ndcg_10",
91+
"value": 0.74095,
92+
"timestamp": "2026-05-01T00:00:00+00:00"
93+
},
94+
{
95+
"task": "SciFact",
96+
"config": "miniLM-bge-reranker-v2-m3",
97+
"metric": "ndcg_10",
98+
"value": 0.73733,
99+
"timestamp": "2026-05-01T00:00:00+00:00"
100+
},
101+
{
102+
"task": "SciFact",
103+
"config": "qwen3-8b-dense-only",
104+
"metric": "ndcg_10",
105+
"value": 0.76474,
106+
"timestamp": "2026-05-01T00:00:00+00:00"
65107
}
66108
]
67109
}

0 commit comments

Comments
 (0)