Skip to content

feat: add seeded Thompson-sampling router policy (#48)#49

Merged
ymxlx merged 1 commit into
ymxlx:mainfrom
Nitjsefnie-OSC:feat/48-thompson-router
Jul 17, 2026
Merged

feat: add seeded Thompson-sampling router policy (#48)#49
ymxlx merged 1 commit into
ymxlx:mainfrom
Nitjsefnie-OSC:feat/48-thompson-router

Conversation

@Nitjsefnie

Copy link
Copy Markdown
Contributor

Closes #48.

Adds a thompson routing policy beside epsilon-greedy (default) and ucb, mirroring the structure of the UCB router from #44 — same POLICIES registry, same per-citizen sample-count aggregation across required_tags, and --explain naming the active policy.

Formulation (Gaussian Thompson sampling): each citizen's combined score is the posterior mean; the width is THOMPSON_EXPLORATION_C / sqrt(n) (c = sqrt(2), matching UCB_EXPLORATION_C), with unplayed citizens (n=0) getting a wider fixed prior (sigma=2.0) so posterior width shrinks monotonically with evidence and cold-start explores everyone broadly. Sample one value per citizen, pick the max, ties break by citizen id.

Determinism (the crux — sampling is stochastic, and the suite must be reproducible): pick_thompson/thompson_sample_values take an injected random.Random — never global random. recommend(seed=) and main() build it from --seed, the exact convention epsilon-greedy already uses. Citizens are sampled in citizen-id order, so the draw sequence (and every sample) is a pure function of the seed, independent of dict order — a given (state, seed) is fully reproducible. The flag-free epsilon-greedy default path is textually untouched and byte-identical across fixed seeds (0/1/7/42/999).

Tests (ThompsonRoutingTest in scripts/test_lessons_routing.py, mirroring UCBRoutingTest): monotonic posterior-width shrink, 20-call seeded determinism, order-independence of the sampling seam, cold-start spread across unplayed arms, exploit-the-leader on ample evidence, empty-input safety, and a decision test pinned to the mean/evidence accounting. That last one bites: dropping the mean from the sample (rng.gauss(score, …)rng.gauss(0.0, …)) flips the chosen arm and fails the test; restored, all 21 pass. references/routing.md gains a Thompson subsection beside UCB and the README policy list is updated.

All CI test scripts pass on both main and this branch, and --policy thompson --seed 7 --explain produces a reproducible pick with the sorted sampled-value breakdown.

Disclosure: this contribution was made with AI assistance (Claude), verified against the repo's full test suite and mutation-tested before submission.

Add `--policy thompson` to route_contract.py alongside the epsilon-greedy
default and the UCB variant. epsilon-greedy stays the default and its routing
decisions remain byte-identical to before; only `--explain` gains the policy
name and, for thompson, the per-citizen sampled posterior values.

Thompson sampling treats each citizen's combined score as the mean of a
Gaussian posterior with standard deviation c/sqrt(n) (c=sqrt(2)); it samples one
value per citizen and picks the max. Unplayed citizens (n=0) get a wide fixed
prior (sigma=2.0) so cold-start spreads work across every untried citizen. Like
epsilon-greedy it is stochastic, so an injected/seeded random.Random threads
through recommend(..., seed=) and --seed; citizens are sampled in id order so a
given (state, seed) is fully reproducible. Model and tradeoff vs epsilon-greedy
and UCB documented in references/routing.md; seeded determinism, cold-start
spread, exploit-on-ample-evidence, and empty-input safety covered in
scripts/test_lessons_routing.py.

Closes ymxlx#48

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Nitjsefnie
Nitjsefnie requested a review from ymxlx as a code owner July 17, 2026 06:36
@vercel

vercel Bot commented Jul 17, 2026

Copy link
Copy Markdown

@Nitjsefnie is attempting to deploy a commit to the yehudamoshe24-6977's projects Team on Vercel.

A member of the Team first needs to authorize it.

@ymxlx ymxlx left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified locally against the research-team example: 21 routing tests pass (7 new Thompson cases), --policy thompson --seed 7 is reproducible across runs, and the flag-free epsilon-greedy default path is byte-identical. Clean mirror of the UCB structure (#44) — injected RNG, citizen-id-ordered sampling for seed-determinism, mutation-pinned decision test, docs + roadmap updated. Thanks for the thorough writeup and the disclosure. Approving.

@ymxlx
ymxlx merged commit de42a9e into ymxlx:main Jul 17, 2026
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[good first issue] Thompson-sampling router policy (--policy thompson)

2 participants