Measurement hardening: deterministic extraction, bounded transcripts, per-model reasoning effort (v5) - #162
Merged
Merged
Conversation
Pin extraction-class ModelRequest calls to temperature=0.0 so E/P1 measurement surfaces stop wandering across identical-binding runs (#154). Bound transcript/identity_as_of to a recent-first default of 40 rows with explicit envelope truncation (#156). Add a per-model REMEMBERSTACK_OPENROUTER_REASONING_EFFORT_MAP override on top of the global effort pin so models that need effort=none can coexist with reasoning models (#155). Update the identity_as_of descriptor for honesty (catalog hash recomputed) and record the transcript bound as a dated design amendment.
…rt map Codex review (REQUEST-CHANGES, 5 MAJOR / 4 MINOR) on the measurement- hardening branch, all findings applied: - Bump every generate-owning component version (:temp0-1 generations): temperature is a generation parameter; outputs made without the pin must not replay under the pinned identity. Pin the p2_analytics label call too. - identity_as_of v2 -> v3 with an optional bounded `limit` parameter (minimum 1) so a truncated history is recoverable through the public surface; upgrade test proves a persisted v2 row is superseded despite ON CONFLICT DO NOTHING. - Protocol RS-LoCoMo-Full-v4 -> v5: the tool-catalog hash changed, dated design note added; new hash 34d2069ae37f. - Forward REMEMBERSTACK_OPENROUTER_REASONING_EFFORT_MAP through Compose x-app (it never reached containers) and report it canonically in the readiness model bindings. - Deterministic entity-transcript truncation boundary (primary-key tie- breaker across both union arms). - AST temperature test discovers all workers/ and spine/ modules and attribute-qualified ModelRequest spellings; effort-map tests assert our error messages and cover malformed env forms. Gate: ruff format/check, pyright 0 errors, pytest 934 passed (incl. Postgres-gated suites). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKENhTLJg1HqhbdwCmmkbc
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measurement hardening: deterministic extraction, bounded transcripts, per-model reasoning effort
Closes #154, #155, #156.
What
#154 — extraction is measurement, not creativity. Every extraction-class
ModelRequest(E0 structurer, E1 prefixer, E2 selection+Claimify, E3normalizer, P1 fact labels, P2 community labels, resolver T4, observation and
supersession adjudicators) pins
temperature=0.0. Gold-span coverage waswandering 7/8 → 4/8 across identical-binding runs. An AST test discovers every
module under
workers/andspine/(fixed lists rot) and fails closed on anyun-pinned site, including attribute-qualified spellings.
#156 — the transcript surface is recent-first bounded with a disclosed cap.
QueryEngine.transcriptkeeps the newestlimitrows (default 40), alwayssets the envelope truncation marker with exact totals, and the
identity_as_ofrecipe (v3) exposes an optionallimit(minimum 1) so atruncated history is recoverable through the public surface. The entity
transcript SQL gains a primary-key tie-breaker so the truncation boundary is
deterministic under batch inserts (same-timestamp rows).
#155 — per-model reasoning effort.
REMEMBERSTACK_OPENROUTER_REASONING_EFFORT_MAP(JSON object, model id → effort) overrides the global pin per model, so
flash-class models can run
nonewhile luna-class models think. Validatedstrictly (allowed literals only, non-empty keys, empty string = unset),
forwarded through Compose
x-app, and reported in canonical form by thereadiness bindings so the effective policy is measurement provenance.
Provenance consequences (review findings, applied)
:temp0-1generation):structurer, prefixer, extractor, normalizer, fact-label, resolver, both
adjudicators, community detector. Temperature is a generation parameter;
outputs made without the pin must not replay/skip under the pinned identity.
identity_as_ofrecipe v2 → v3:ON CONFLICT DO NOTHINGwould otherwise pinexisting deployments to the pre-truncation descriptor. Upgrade test proves a
persisted v2 row is superseded on re-seed.
limitparameter), soRS-LoCoMo-Full-v5with a dated design note. The v4glm-4.7-flash smoke scores are not comparable forward.
34d2069ae37fabf033d2b1f0fae2ed9e7c1ad5c3f6e1fd14b2971344cd89c3a6.Review trail
Grok-4.5 implemented the base branch; Codex (gpt-5.6-sol, xhigh) reviewed with
verdict REQUEST-CHANGES (5 MAJOR, 4 MINOR); this revision applies all nine
findings. Final review and adjustments by Claude. Consciously omitted: a
version-constants regression test (a hardcoded string-inequality test rots
instantly; the bump discipline stays review-enforced).
Gate
ruff format --check+ruff check+pyright(0 errors) + full pytestincluding Postgres-gated suites — results in the checks below.