Skip to content

diversity_rerank_dev.md

Latest commit

 

History

History
106 lines (92 loc) · 6.89 KB

diversity_rerank_dev.md

File metadata and controls

106 lines (92 loc) · 6.89 KB
Raw

TRACE Diversity Reranking - Dev Report

Selection is performed once on median seed 20260715. Seeds 20260714 and 20260716 are validation-only and do not retune parameters.

Frozen configuration

  • Raw candidate pool: top 200
  • Reranked depth: top 100
  • Max NDCG@20 relative loss: 2.0%
  • Max Recall@50 relative loss: 2.0%
  • Max warm-tail Recall@50 relative loss: 2.0%
  • Max Recommendation Gini@50 relative increase: 0.5%
  • Workbench default (balanced): {"candidate_pool_size": 200, "catalog_novelty_weight": 0.6, "entropy_power": 1.0, "max_diversity_weight": 0.2, "popularity_smoothing": 1.0, "preserve_top_n": 1, "profile_temperature": 1.0, "responsibility_temperature": 0.1}
  • Optional cold exploration: {"candidate_pool_size": 200, "catalog_novelty_weight": 1.0, "entropy_power": 0.75, "max_diversity_weight": 0.6, "popularity_smoothing": 1.0, "preserve_top_n": 1, "profile_temperature": 1.0, "responsibility_temperature": 0.1}

Balanced three-seed validation

Seed Role Raw NDCG@20 Reranked NDCG@20 Raw Recall@50 Reranked Recall@50 Raw Coverage@50 Reranked Coverage@50
20260715 selection 0.012161 0.012182 0.031787 0.032153 0.709343 0.710189
20260714 validation_only 0.012203 0.012330 0.033158 0.033650 0.709872 0.709766
20260716 validation_only 0.008387 0.008455 0.022068 0.022395 0.717913 0.718337

Policy trade-offs (three-seed mean)

Diagnostic Raw Balanced default Cold exploration
NDCG@20 0.010917 0.010989 0.011707
Recall@50 0.029004 0.029399 0.033040
Catalog coverage@50 0.712376 0.712764 0.716573
Average novelty@50 17.142665 17.230533 17.862339
Recommendation Gini@50 0.870321 0.871035 0.877208
Warm-tail recall@50 0.043243 0.042534 0.036322
Cold-item recall@50 0.027863 0.028622 0.035179

Balanced is the workbench default because it bounds warm-tail recall loss and concentration growth. Cold exploration is retained only as an optional diagnostic policy for scenarios that explicitly prioritize new/cold discovery.

Two-stage selection grid (seed 20260715 only)

Trial Stage max lambda novelty entropy power NDCG@20 Recall@50 Coverage@50 Feasible
1 coarse 0.000 0.000 1.000 0.012161 0.031787 0.709343 yes
2 coarse 0.050 0.000 0.750 0.012172 0.031803 0.709237 yes
3 coarse 0.050 0.000 1.250 0.012169 0.031796 0.709237 yes
4 coarse 0.050 0.350 0.750 0.012172 0.031891 0.709555 yes
5 coarse 0.050 0.350 1.250 0.012173 0.031855 0.709343 yes
6 coarse 0.100 0.000 0.750 0.012177 0.031803 0.708920 yes
7 coarse 0.100 0.000 1.250 0.012173 0.031796 0.709449 yes
8 coarse 0.100 0.350 0.750 0.012169 0.032011 0.709872 yes
9 coarse 0.100 0.350 1.250 0.012181 0.031892 0.709343 yes
10 coarse 0.150 0.000 0.750 0.012176 0.031814 0.708920 yes
11 coarse 0.150 0.000 1.250 0.012181 0.031811 0.709237 yes
12 coarse 0.150 0.350 0.750 0.012184 0.032065 0.709872 yes
13 coarse 0.150 0.350 1.250 0.012170 0.031987 0.709555 yes
14 refinement 0.200 0.350 0.500 0.012191 0.032253 0.709978 yes
15 refinement 0.200 0.350 0.750 0.012188 0.032109 0.709872 yes
16 refinement 0.200 0.350 1.000 0.012179 0.032049 0.709872 yes
17 refinement 0.200 0.600 0.500 0.012239 0.032481 0.711036 no
18 refinement 0.200 0.600 0.750 0.012201 0.032276 0.709872 no
19 refinement 0.200 0.600 1.000 0.012182 0.032153 0.710189 yes
20 refinement 0.200 1.000 0.500 0.012248 0.033066 0.711882 no
21 refinement 0.200 1.000 0.750 0.012225 0.032754 0.711036 no
22 refinement 0.200 1.000 1.000 0.012207 0.032349 0.710824 no
23 refinement 0.300 0.350 0.500 0.012217 0.032444 0.710613 no
24 refinement 0.300 0.350 0.750 0.012212 0.032311 0.709978 no
25 refinement 0.300 0.350 1.000 0.012181 0.032134 0.709872 yes
26 refinement 0.300 0.600 0.500 0.012245 0.032903 0.710824 no
27 refinement 0.300 0.600 0.750 0.012221 0.032603 0.710718 no
28 refinement 0.300 0.600 1.000 0.012205 0.032321 0.710507 no
29 refinement 0.300 1.000 0.500 0.012328 0.033823 0.712835 no
30 refinement 0.300 1.000 0.750 0.012245 0.033180 0.711459 no
31 refinement 0.300 1.000 1.000 0.012223 0.032827 0.710930 no
32 refinement 0.400 0.350 0.500 0.012248 0.032694 0.710718 no
33 refinement 0.400 0.350 0.750 0.012209 0.032426 0.710189 no
34 refinement 0.400 0.350 1.000 0.012195 0.032263 0.709343 no
35 refinement 0.400 0.600 0.500 0.012300 0.033396 0.711882 no
36 refinement 0.400 0.600 0.750 0.012229 0.032921 0.710613 no
37 refinement 0.400 0.600 1.000 0.012212 0.032599 0.710507 no
38 refinement 0.400 1.000 0.500 0.012485 0.034664 0.714739 no
39 refinement 0.400 1.000 0.750 0.012258 0.033664 0.712517 no
40 refinement 0.400 1.000 1.000 0.012220 0.033214 0.711142 no
41 refinement 0.600 0.350 0.500 0.012305 0.033320 0.712306 no
42 refinement 0.600 0.350 0.750 0.012249 0.032799 0.710824 no
43 refinement 0.600 0.350 1.000 0.012201 0.032561 0.710189 no
44 refinement 0.600 0.600 0.500 0.012489 0.034681 0.714316 no
45 refinement 0.600 0.600 0.750 0.012281 0.033609 0.712940 no
46 refinement 0.600 0.600 1.000 0.012241 0.033078 0.711142 no
47 refinement 0.600 1.000 0.500 0.012807 0.036992 0.714633 no
48 refinement 0.600 1.000 0.750 0.012510 0.035017 0.715691 no
49 refinement 0.600 1.000 1.000 0.012275 0.033950 0.713152 no

Search boundary

  • Boundary status below applies to optional cold exploration.
  • Aggressive selected maximum diversity weight: 0.600
  • Highest tested maximum diversity weight: 0.600
  • Selected at upper weight boundary: true
  • The deterministic refinement grid is terminal; no iterative boundary chasing is performed.

Protocol safeguards

  • Existing TRACE checkpoints are loaded strictly; no retraining occurs.
  • Daily as-of availability and strictly prior positive history are rebuilt.
  • Seen items are masked before raw top-200 retrieval.
  • Test pairs and small-matrix Test preferences are not read.
  • Zero-diversity top-100 replay must match saved Dev artifacts exactly.