feat: S1 relevance-scoped entity_detail selection (A2a lever (b)) -- staged S1a recover-only -> S1b drop-filler

[daviburg]

Summary

Implement S1 of lever (b) / A2a: replace the arbitrary cap-6 entity_detail
truncation with relevance-scoped selection, reusing the existing #233
_select_context_aware_entities signal that S0 already shadowed. This is the
behaviour-changing slice that follows the S0 shadow (#494, verdict GO on the
full 344-turn replay at HEAD 27dfc88).

Design: docs/design-a2a-s1-relevance-scoped-detail.md (read it first; all
line citations below are against origin/main 27dfc88).

Parent epic: #477 (token-demand stabilization). S0 umbrella / parent: #494.

---

The key reframe (be honest about what S1 is)

Per the S0 arithmetic, across the 344-turn run relevance adds 245 detail
calls (the cap-dropped-but-relevant recoveries) and removes 232 detail
calls (the would-drop off-scene filler): net ~ +13 detail calls run-wide
= ~ +0.04 calls/turn.

Therefore S1 is a QUALITY + DECOUPLING lever, NOT a token-reduction lever.
It recovers real dropped updates and makes calls/turn track scene size
instead of an arbitrary cap. Its mean per-turn token delta is net ~flat and
marginally positive (it spends a few tokens to recover updates).

Do NOT credit S1 with token savings. The token bound is lever (c) / S2-S3
(PC permanent-web cap #495 + checkpoint context reset #496). S1 must not claim
to flatten the 64-151 tok/turn total-input slope -- that slope is dominated by
per-call body growth (carried context), which S1 does not touch.

---

WHAT changes (the wiring)

Replace the cap-6 truncation block at tools/semantic_extraction.py
~L4048-4076 with relevance-scoped selection, before
_partition_detail_tasks (L4181) so L2 (batch_entity_detail) composes
unchanged on the selected set.

Flag ON (S1) selection rule -- the EXACT rule S0 shadowed:

all_entities       = flatten(catalogs)
_ordered, prio_ids = _select_context_aware_entities(all_entities, turn_text,
                                                     current_turn, _DEFAULT_RECENCY_WINDOW)
detail_set = [ (ref, entry) for (ref, entry) in _entity_tasks
               if is_pc(ref)                     # hard floor: PC always
                  or ref.get("is_new", True)     # hard floor: new always
                  or get_entity_id(ref) in prio_ids ]   # relevance tiers
detail_set = _apply_relevance_ceiling(detail_set, prio_ids, _ordered)  # ceiling, relevance-ordered tail drop
_entity_tasks = detail_set

• Reuse _select_context_aware_entities (tools/catalog_merger.py L597, feat: context-aware entity selection for discovery prompts (location, relationships, co-occurrence) #233)
  verbatim -- no new scorer, no new word list (Rule 9).
• Keep the S0 shadow block (L4078-4083) ON during S1 runs so live
  would_select vs actual is auditable (live-vs-shadow risk).
• PC stays solo, new stays solo -- unchanged; the floor mirrors L2 tiering.

Default-OFF flag (the A/B control)

New gate context_optimizations.relevance_scoped_detail, default OFF,
strict-bool parse (mirroring relevance_shadow_logging /
batch_entity_detail / relationship_type_tiering).

Flag-OFF MUST be byte-identical to the current cap-6 path (the cap block
runs unchanged, S1 branch never entered, no shadow/metric change). This is the
A/B control: same commit, flag OFF.

Sub-mode relevance_scoped_detail.mode = "recover" | "strict" selects S1a vs
S1b (see staging).

---

Staging: S1a (recover-only) -> S1b (drop filler)

No single PR both recovers and drops. Recovery (pure upside) is proved
first, then the drop is proved against a control that already has the recovery.

• S1a -- recover-only (generous bound). detail_set = relevance_set UNION cap6_set, ceiling = p95 (~10). Adds the 245 cap-dropped-but-relevant
  updates, drops NOTHING currently detailed. Pure quality gain, lowest risk
  (it can only add). This is the safe first slice.
• S1b -- tighten. detail_set = relevance_set (drops the 232 off-scene
  filler relevance would not pick), same ceiling. Default OFF, depends on S1a
  green. Brings tokens back to net ~flat.
  If S1b's retention gate is red: keep S1a (quality win), abandon S1b
  (filler drop). Clean, safe fallback.

S1a and S1b are checklist items inside this one issue (S1b may be split to its
own PR since it depends on S1a being green). Both are disjoint from lever (c)
(S2/S3) and parallel-safe with them.

---

THE GATE (critical -- the validation discipline)

S1 is gated on a CHEAP QUALITY A/B -- an update-retention diff vs a cap-6
control on a matched replay pair -- NOT the 60h paired TOKEN A/B (#488).

The token A/B is explicitly SKIPPED for S1 because pre-check (i)
(effect > noise) FAILS on the token metric: S1's projected token delta is
~ +0.04 calls/turn -- within run noise (~2.4 %; noise floor ~5 tok/call) and
pointing the wrong direction (a marginal increase, not a reduction).
Spending 60h GPU to confirm a flat token delta would repeat the L2 mistake
(effect within noise), inverted. No agent should later spend the 60h token
A/B on S1.

The quality A/B's effect is large and countable (245 recovered updates,
232 dropped filler) -- orders of magnitude above noise -- and reflects S1's
real value. The calls/turn-vs-catalog-size plateau is already proven by
S0 (would_select share fell 7.3% -> 5.4% as catalog grew ~7 -> 126), so no
new slope run is needed for S1. The expensive whole-curve total-token slope->0
test is deferred to after lever (c) / S2-S3, where the total-token slope
actually moves.

---

Bound (ceiling + floor)

• Ceiling = S0 would_select p95 ~= 10 (telemetry-derived, caps the
  max-18 climax-scene spike; binds on < 5% of turns at p50 = 3). On bind, drop
  the lowest-relevance tail via the _ordered ranking (strictly better
  than today's lowest-confidence cap-6 drop). The existing
  _MAX_DETAIL_ENTITIES_PER_TURN = 6 (L1163) is retained as a hard safety
  ceiling.
• Floor = PC + new + every mentioned (tier-1) entity are force-kept.
  calls/turn >= |new| + |mentioned| + 1(PC).
• The one genuinely new knob = the ceiling (relevance_detail_max_per_turn,
  default ~10 from S0 p95). The speculative relevance_detail_threshold scalar
  is explicitly NOT added -- selection is discrete tier membership, not a
  scalar score (Rule 10 debt avoidance).

---

Acceptance criteria

S1a (recover-only -- do this first)

1. relevance_scoped_detail flag, default OFF, byte-identical to the current
   cap-6 path when OFF (control test in CI).
2. Flag ON details the relevance set ({PC, new} UNION priority_ids) and
   recovers the cap-dropped-but-relevant updates (the 245), dropping nothing
   currently detailed in S1a (detail_set = relevance_set UNION cap6_set,
   ceiling = p95 ~10).
3. The recovered updates actually land in the catalog: an update-retention
   diff vs the cap-6 control shows the gained updates (retention strictly

   = control).

4. No coreference regression / no duplicate minting (the feat: adaptive pressure-gated context compression #468 dedup audit
   on the variant run shows no increase over control).
5. The non-blind metric records the change: per-turn entity_detail
   calls/tokens via fix: entity_detail prompt-token instrumentation is blind to compaction (raw_tokens captured post-compression) #484 + ab_paired_score --per-turn (feat: paired multi-run A/B scoring on the entity_detail token metric (#487) #488/feat: batch entity_detail calls to collapse template+turn duplication (L2, #491) #493) reflect the
   new selection on the SENT path.
6. S0 shadow logging stays available so a S1 run can compare live
   would_select vs the actual (relevance-driven) set.
7. Tests: (a) flag-off byte-identity, (b) recover-set correctness
   (relevance_set UNION cap6_set, ceiling, lowest-relevance-tail drop on
   bind, floor invariant 100%), (c) the quality update-retention check.
8. Docs (Rule 8): update docs/architecture.md (new relevance-scoped
   detail-selection layer) and docs/roadmap.md (epic Epic: Stabilize per-turn extraction token demand at quality parity (bounded growth over 300+ turns) #477 / S1 progress) in
   the implementing PR.

S1b (tighten -- after S1a's retention check passes)

1. detail_set = relevance_set (drop the 232 filler), same ceiling.
2. Missed-update diff vs the S1a/cap-6 control = 0 material: drops only
   entities a retention check confirms are low-value off-scene filler (no
   real update lost). Net token delta returns to ~flat.
3. Coreference recall >= control; zero new duplicates.
4. If the retention gate is red: keep S1a, abandon S1b (documented
   fallback).

---

Dependencies / notes

• S1a is the safe first slice; S1b follows only after S1a's retention check
  passes.
• Reuses feat: context-aware entity selection for discovery prompts (location, relationships, co-occurrence) #233 (_select_context_aware_entities), builds on feat: relevance-scoped entity_detail selection (A2a) -- bound calls/turn (epic #477, lever b) #494 (S0 shadow,
  GO). Parallel-safe with lever (c) (S2/S3: feat: cap+age PC permanent-bond web -- extend L1 type-tiering (epic #477, lever c-i) #495, feat: checkpoint context reset/rebuild with identity-anchor floor (epic #477, lever c-ii) #496).
• Token parity is NOT a gate (S1 is token-neutral by design).

Design doc: docs/design-a2a-s1-relevance-scoped-detail.md

Part of epic #477. Child of #494.