feat: add seeded Thompson-sampling router policy (#48)#49
Merged
Conversation
Add `--policy thompson` to route_contract.py alongside the epsilon-greedy default and the UCB variant. epsilon-greedy stays the default and its routing decisions remain byte-identical to before; only `--explain` gains the policy name and, for thompson, the per-citizen sampled posterior values. Thompson sampling treats each citizen's combined score as the mean of a Gaussian posterior with standard deviation c/sqrt(n) (c=sqrt(2)); it samples one value per citizen and picks the max. Unplayed citizens (n=0) get a wide fixed prior (sigma=2.0) so cold-start spreads work across every untried citizen. Like epsilon-greedy it is stochastic, so an injected/seeded random.Random threads through recommend(..., seed=) and --seed; citizens are sampled in id order so a given (state, seed) is fully reproducible. Model and tradeoff vs epsilon-greedy and UCB documented in references/routing.md; seeded determinism, cold-start spread, exploit-on-ample-evidence, and empty-input safety covered in scripts/test_lessons_routing.py. Closes ymxlx#48 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
@Nitjsefnie is attempting to deploy a commit to the yehudamoshe24-6977's projects Team on Vercel. A member of the Team first needs to authorize it. |
ymxlx
approved these changes
Jul 17, 2026
ymxlx
left a comment
Owner
There was a problem hiding this comment.
Verified locally against the research-team example: 21 routing tests pass (7 new Thompson cases), --policy thompson --seed 7 is reproducible across runs, and the flag-free epsilon-greedy default path is byte-identical. Clean mirror of the UCB structure (#44) — injected RNG, citizen-id-ordered sampling for seed-determinism, mutation-pinned decision test, docs + roadmap updated. Thanks for the thorough writeup and the disclosure. Approving.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #48.
Adds a
thompsonrouting policy besideepsilon-greedy(default) anducb, mirroring the structure of the UCB router from #44 — samePOLICIESregistry, same per-citizen sample-count aggregation acrossrequired_tags, and--explainnaming the active policy.Formulation (Gaussian Thompson sampling): each citizen's combined
scoreis the posterior mean; the width isTHOMPSON_EXPLORATION_C / sqrt(n)(c =sqrt(2), matchingUCB_EXPLORATION_C), with unplayed citizens (n=0) getting a wider fixed prior (sigma=2.0) so posterior width shrinks monotonically with evidence and cold-start explores everyone broadly. Sample one value per citizen, pick the max, ties break by citizen id.Determinism (the crux — sampling is stochastic, and the suite must be reproducible):
pick_thompson/thompson_sample_valuestake an injectedrandom.Random— never globalrandom.recommend(seed=)andmain()build it from--seed, the exact convention epsilon-greedy already uses. Citizens are sampled in citizen-id order, so the draw sequence (and every sample) is a pure function of the seed, independent of dict order — a given(state, seed)is fully reproducible. The flag-free epsilon-greedy default path is textually untouched and byte-identical across fixed seeds (0/1/7/42/999).Tests (
ThompsonRoutingTestinscripts/test_lessons_routing.py, mirroringUCBRoutingTest): monotonic posterior-width shrink, 20-call seeded determinism, order-independence of the sampling seam, cold-start spread across unplayed arms, exploit-the-leader on ample evidence, empty-input safety, and a decision test pinned to the mean/evidence accounting. That last one bites: dropping the mean from the sample (rng.gauss(score, …)→rng.gauss(0.0, …)) flips the chosen arm and fails the test; restored, all 21 pass.references/routing.mdgains a Thompson subsection beside UCB and the README policy list is updated.All CI test scripts pass on both
mainand this branch, and--policy thompson --seed 7 --explainproduces a reproducible pick with the sorted sampled-value breakdown.Disclosure: this contribution was made with AI assistance (Claude), verified against the repo's full test suite and mutation-tested before submission.