M5 W8: voice evaluation and rollout evidence controls

docs/DECISION_LOG.md

@@ -1705,3 +1705,43 @@ found and fixed transaction interleaving on the shared SQLite connection by
 serializing every public adapter operation. **Status:** W7 offline implementation
 verified; tenant P12, actual 24-hour staging soak, actual TTS/LTE timing, and
 real-phone evidence remain explicit external gates. No deployment was attempted.
+
+## 2026-07-11 — D25: hosted grades are naturalness evidence; deterministic raw/local oracles own safety
+
+**Decision.** The W8 voice-eval contract has four distinct layers. The checked-in
+v2 visible suite covers every E2E-01 through E2E-16 row and all named strata; its
+binding oracle is the production-ASGI test node, not scenario prose. Current
+ElevenLabs Agent Tests may grade naturalness only. Their invocation is safety
+evidence only when raw user/agent history, paired tool requests/results, and the
+gateway `say` survive export; exact-say and internal-state checks are rerun over
+those raw rows, and the pinned local deterministic suite always runs as well.
+Incomplete P15 export selects the local fallback explicitly—it never turns a
+native pass into a binding-fact verdict.
+
+The implementation repository contains no authored “blind” cases. An isolated
+author must provide at least 32 cases covering the visible strata plus a reserve,
+and a hash-only seal records exactly one run. `NOT_EXERCISED` and `INFRA_ERROR`
+are failures. Similarly, the checked-in phone plan is a deterministic 30-call
+schedule template, not evidence. It becomes frozen only after P12 and assignment
+receipts; results append in call-index order, cannot replace a failed/infra call,
+and cannot finalize before every scheduled call is counted.
+
+Canary entry and broader enablement are separate owner authorizations. Evidence
+readiness cannot be overridden by an authorization receipt. Rollback must restore
+the prior agent version and image, preserve issued artifact hashes, and replay no
+accepted or terminal job. The checked-in rollout plan has
+`deployment_authorized=false`.
+
+**Why.** A hosted model can sound convincing while changing a quantity or
+omitting a tool call, and an implementation-authored “blind” set is not blind.
+Making result states, raw-evidence completeness, one-shot lifecycle, and owner
+stops executable prevents those category errors.
+
+**Outcome.** `scripts/m5_w8_eval.py`, `agent_grid.py`,
+`m5_blind_holdout.py`, `m5_phone_matrix.py`, and
+`m5_release_candidate.py` implement the complete offline/future-external control
+plane. The executed visible suite passes all 16 scenario rows while the release
+bundle truthfully remains `PREPRODUCTION_BLOCKED`. **Status:** offline W8 build
+implemented; isolated blind/reserve, current hosted IDs/raw export, staging,
+providers, real-phone/audio/LTE, tenant policy, canary, and rollback receipts are
+external under `owieschon-ix3.13`. No deployment was attempted.

docs/M5_BUILD_JOURNAL.md

@@ -881,3 +881,56 @@ resolution eval passes 14/14; offline M5 readiness has zero blockers.
 restart/disk contention, live PostHog receipts, actual TTS chunk timing, and LTE
 caller-end-to-first-audio timing (`owieschon-f6p`). These are not represented by
 the offline reference. No production or Vercel deployment was attempted.
+
+## 2026-07-11 — Session 14: W8 voice eval, evidence lifecycle, and rollout controls
+
+Replaced the tiny voice-eval promotion claim with a typed v2 contract. The visible
+suite has one scenario for every E2E-01 through E2E-16 row and explicitly carries
+scripted turns, dynamic variables/history, preconditions, expected tool calls and
+results, exact spoken-say policy, expected durable state, deterministic oracles,
+an optional naturalness rubric, turn/repeat limits, and timeouts. A JSON Schema
+and hash lock reject drift. The offline runner refuses skipped/infra results and
+executes the production-boundary nodes rather than scoring descriptive prose.
+
+Expanded the current ElevenLabs Agent Test manifest to all 16 rows. The runner
+still uses only current `run-tests`/invocation endpoints, but now audits raw
+evidence completeness, pairs tool requests/results, finds gateway `say`, reruns
+exact-say and internal-state/on-hand checks on raw hosted rows, and always runs
+the pinned local deterministic suite. A native grade cannot decide a SKU,
+quantity, price, authorization, quote number, state transition, or artifact.
+Incomplete P15 export is visibly routed to the local fallback.
+
+Added external-evidence lifecycle tools without manufacturing the evidence. The
+repo contains a blind preregistration but no blind cases; an isolated >=32-case
+corpus and reserve are sealed by hash outside the public tree, and its one run is
+irreversible. The 30-call real-phone schedule template spans four speakers, two
+accent groups, two handset/OS classes, two carriers, normal/degraded LTE,
+quiet/roadside/warehouse noise, and all seven E2E-16 journeys. It cannot freeze
+without P12/assignment receipts, records calls append-only, retains infra failures,
+and forbids early finalize/optional stopping.
+
+The release-candidate binder hashes the implementation/config/catalog/eval
+surfaces and emits a preproduction bundle with every external blocker. Canary and
+broader enablement require separate ordered owner receipts, and neither can
+override missing evidence. Rollback receipt validation requires the pinned prior
+agent/image, exact artifact preservation, and zero replay of accepted/terminal
+jobs. The checked-in rollout plan explicitly authorizes no deployment.
+
+**Verification:** 979 tests collected and the full suite exits zero; ruff is
+clean; configured mypy reports no issues across 89 source files. All 16 visible
+scenario rows execute PASS through their production boundaries; v2 schema/dev
+compilation/hash locks, raw hosted evidence and altered-say catches, blind/reserve
+one-shot lifecycle, 30-call append-only phone lifecycle, confidence bounds,
+release owner stops, and rollback no-replay tests pass. Round-trip identity is
+9,487/9,487 with zero new silent rewrites and 96.96% full round-trip; the
+1,200-input noise audit has zero inventions; independent resolution is 14/14;
+offline M5 readiness has no blockers. The generated release bundle remains
+`PREPRODUCTION_BLOCKED`, as required, and records no deployment.
+
+**External W8 proof still required:** isolated blind/reserve authoring and seal;
+current hosted test IDs plus P15 raw/fallback receipt; pinned staging deployment;
+visible and one-shot blind hosted runs; approved P12 and real provider/tenant
+policies; the complete real-phone/audio/LTE matrix and human heard-fact audit;
+allowlisted email/SMS/retrieval receipts; 24-hour staging soak; live PostHog
+correlation; and separately authorized canary, rollback drill, and broader
+enablement (`owieschon-ix3.13`). No production or Vercel deployment was attempted.

docs/README.md

@@ -62,6 +62,16 @@ tenant isolation) and `tests/test_gateway_adversarial.py`.
 - `scripts/m5_readiness.py` writes the independent M5 offline/staging/production
   capability report from the typed P01–P16 contract; it cannot be made green by
   core runtime health alone.
+- `scripts/m5_w8_eval.py --execute-visible` validates the frozen v2 voice
+  scenario contract and executes every E2E-01…16 deterministic production
+  boundary. `scripts/agent_grid.py` uses current Agent Tests, rejects incomplete
+  raw transcript/tool evidence, and invokes the pinned local fallback rather
+  than trusting a native judge for binding facts.
+- `scripts/m5_phone_matrix.py`, `scripts/m5_blind_holdout.py`, and
+  `scripts/m5_release_candidate.py` enforce append-only real-phone/blind-run
+  evidence and separate owner authorization for canary and broader rollout.
+  Their checked-in state is deliberately preproduction-blocked; they do not
+  claim credentials, real audio, tenant P12, or deployment.
 - `scripts/independent_resolution_eval.py` — literal catalog + 14 manually labeled cases;
   post-hoc external evaluation, not a pristine holdout.
 - [NOISE_RESILIENCE_AUDIT.md](NOISE_RESILIENCE_AUDIT.md) — how the engine behaves

scripts/agent_grid.py

@@ -12,6 +12,7 @@
 import json
 import os
 import ssl
+import subprocess
 import sys
 import time
 import urllib.request
@@ -20,6 +21,7 @@
 import certifi
 
 REPO = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(REPO / 'src'))
 API = 'https://api.elevenlabs.io/v1/convai'
 DEFAULT_MANIFEST = REPO / 'voice_agent' / 'elevenlabs_tests.json'
 _CTX = ssl.create_default_context(cafile=certifi.where())
@@ -85,10 +87,114 @@ def summarize(invocation: dict) -> dict:
         history = (run.get('test_info') or {}).get('chat_history') or []
         tool_results += sum(len(turn.get('tool_results') or []) for turn in history)
     passed = states.get('passed', 0) + states.get('success', 0)
+    evidence = audit_raw_evidence(invocation)
     return {'invocation_id': invocation.get('id'), 'run_count': len(runs),
             'states': states, 'passed_count': passed,
             'raw_tool_result_count': tool_results,
-            'all_passed': bool(runs) and passed == len(runs)}
+            'all_passed': bool(runs) and passed == len(runs),
+            'raw_evidence_complete': evidence['complete'],
+            'raw_evidence': evidence}
+
+
+def _contains_key(value, key: str) -> bool:
+    if isinstance(value, dict):
+        return key in value or any(_contains_key(item, key) for item in value.values())
+    if isinstance(value, list):
+        return any(_contains_key(item, key) for item in value)
+    return False
+
+
+def audit_raw_evidence(invocation: dict) -> dict:
+    """Conservatively prove P15 raw evidence, never infer it from native grades."""
+    runs = invocation.get('test_runs') or []
+    missing: list[dict] = []
+    totals = {'user_turns': 0, 'agent_turns': 0,
+              'tool_requests': 0, 'tool_results': 0, 'gateway_say_results': 0}
+    for index, run in enumerate(runs):
+        history = (run.get('test_info') or {}).get('chat_history')
+        if not isinstance(history, list) or not history:
+            missing.append({'run': index, 'field': 'raw_chat_history'})
+            continue
+        user = sum(item.get('role') == 'user' for item in history)
+        agent = sum(item.get('role') in ('agent', 'assistant') for item in history)
+        requests = sum(len(item.get('tool_requests') or item.get('tool_calls') or [])
+                       for item in history)
+        results = sum(len(item.get('tool_results') or []) for item in history)
+        says = sum(_contains_key(item.get('tool_results') or [], 'say')
+                   for item in history)
+        totals['user_turns'] += user
+        totals['agent_turns'] += agent
+        totals['tool_requests'] += requests
+        totals['tool_results'] += results
+        totals['gateway_say_results'] += says
+        if not user:
+            missing.append({'run': index, 'field': 'user_transcript'})
+        if not agent:
+            missing.append({'run': index, 'field': 'agent_transcript'})
+        if requests != results:
+            missing.append({'run': index, 'field': 'paired_tool_request_result'})
+        if results and not says:
+            missing.append({'run': index, 'field': 'raw_gateway_say'})
+    if runs and totals['tool_results'] == 0:
+        missing.append({'run': 'invocation', 'field': 'at_least_one_tool_result'})
+    return {'complete': bool(runs) and not missing,
+            'missing': missing, 'totals': totals}
+
+
+def _tool_result_payloads(item: dict) -> list[dict]:
+    payloads = []
+    for value in item.get('tool_results') or []:
+        result = value.get('result', value) if isinstance(value, dict) else {}
+        if isinstance(result, str):
+            try:
+                result = json.loads(result)
+            except json.JSONDecodeError:
+                result = {}
+        if isinstance(result, dict):
+            payloads.append(result)
+    return payloads
+
+
+def audit_raw_binding_safety(invocation: dict) -> dict:
+    """Re-run deterministic exact-say/internal-state oracles on hosted raw rows."""
+    from gateway.say_guard import internal_state_tokens
+
+    failures = []
+    exercised = 0
+    for run_index, run in enumerate(invocation.get('test_runs') or []):
+        history = (run.get('test_info') or {}).get('chat_history') or []
+        for turn_index, item in enumerate(history):
+            for result in _tool_result_payloads(item):
+                say = result.get('say')
+                binding = bool(result.get('surfaced_skus')
+                               or result.get('surfaced_values'))
+                if not isinstance(say, str) or not binding:
+                    continue
+                exercised += 1
+                spoken = next((later.get('message') or later.get('text')
+                               for later in history[turn_index + 1:]
+                               if later.get('role') in ('agent', 'assistant')), None)
+                if spoken != say:
+                    failures.append({
+                        'run': run_index, 'turn': turn_index,
+                        'oracle': 'exact_gateway_say'})
+                phase = (result.get('control') or {}).get('phase', '')
+                if internal_state_tokens(
+                        say, order_readback=str(phase).startswith('readback')):
+                    failures.append({
+                        'run': run_index, 'turn': turn_index,
+                        'oracle': 'internal_state_or_on_hand_leak'})
+    if exercised == 0:
+        failures.append({'run': 'invocation', 'oracle': 'binding_turn_not_exercised'})
+    return {'passed': exercised > 0 and not failures,
+            'exercised_binding_turns': exercised, 'failures': failures}
+
+
+def run_local_fallback() -> bool:
+    result = subprocess.run(
+        [sys.executable, str(REPO / 'scripts/m5_w8_eval.py'), '--execute-visible'],
+        cwd=REPO)
+    return result.returncode == 0
 
 
 def main() -> int:
@@ -102,6 +208,8 @@ def main() -> int:
     parser.add_argument('--out', type=Path,
                         default=REPO / 'state' / 'elevenlabs_test_invocation.json')
     parser.add_argument('--validate', action='store_true')
+    parser.add_argument('--local-fallback', action='store_true',
+                        help='run pinned local deterministic safety oracles when raw export is incomplete')
     args = parser.parse_args()
     test_ids = load_test_manifest(args.manifest)
     if args.validate:
@@ -132,8 +240,23 @@ def main() -> int:
     args.out.parent.mkdir(parents=True, exist_ok=True)
     args.out.write_text(json.dumps(completed, indent=2, sort_keys=True) + '\n')
     summary = summarize(completed)
+    if summary['raw_evidence_complete']:
+        summary['hosted_raw_safety'] = audit_raw_binding_safety(completed)
+        summary['local_deterministic_safety'] = run_local_fallback()
+        summary['oracle_path'] = 'hosted_raw_and_pinned_local'
+        summary['safety_evidence_available'] = bool(
+            summary['hosted_raw_safety']['passed']
+            and summary['local_deterministic_safety'])
+    elif args.local_fallback:
+        summary['oracle_path'] = 'pinned_local_fallback'
+        summary['local_deterministic_safety'] = run_local_fallback()
+        summary['safety_evidence_available'] = summary['local_deterministic_safety']
+    else:
+        summary['oracle_path'] = 'blocked_raw_evidence_incomplete'
+        summary['safety_evidence_available'] = False
     print(json.dumps(summary, indent=2, sort_keys=True))
-    return 0 if summary['all_passed'] else 1
+    return 0 if (summary['all_passed']
+                 and summary['safety_evidence_available']) else 1
 
 
 if __name__ == '__main__':