P2 — LLM seam & extraction

[foae]

Goal: provider-agnostic vision extraction with fail-closed metering, gated by a real eval. Spec: design/plays-plan.md §4, design/plays-product.md §4.1.

• Deps: ai, @ai-sdk/openai, sharp; verify ai v5 zod-4 generateObject support and OpenAI strict-output .nullable() (not .optional()) constraint
• PlayAnalyzer seam + injected fake; image downscale via sharp (off the main event loop)
• PlayExtraction zod schema, pinned per product §4.1: per-leg positions with stable position_id, side (quantity always positive), full expiry, ×100 convention, opened_at, currency + prompt v1 (versioned, under plays/prompts/)
• Deterministic validation: three-outcome ticker check (committed known-non-equity list), P&L arithmetic cross-check, confidence derived from the deterministic pass
• markPlay(extraction, quotes) stub + unit tests — the compile-checked P2↔P5 pin
• Metering fail-closed: zero/missing price refuses dispatch; daily cap summed from DB cost_usd rows (UTC day); worst-case pre-dispatch reservation via max_output_tokens; unit tests cover all-zero-price and restart cases
• Per-stage atomicity: child row + status advance + cost in one transaction; LLM call outside any tx (P9)
• Eval: fixtures/plays/ ≥30 stratified, redacted screenshots, hand-labeled; machine-scored per-field; near-perfect thresholds on marking-critical fields (ticker/side/quantity/strike/expiry)
• Ops that must not wait for P6: spend + queue-depth logging; OPENAI_API_KEY in deploy/v2/.env.example; stop_grace_period in compose
• Model + budget decision happens here (deferred by owner 2026-08-18)

Gate: eval thresholds met (hand-judged sample spot-checked by owner); budget behavior proven incl. fail-closed cases; live queue extracts real plays.

[foae]

P2 progress — live extraction PROVEN (2026-08-19).

Code landed across dcadbd0→aed3b15: pinned extraction schema + markPlay P2↔P5 pin + deterministic validation; PlayAnalyzer seam; fail-closed metering (persisted pre-dispatch reservations, DB-summed daily cap); media_ready → extracted queue stage; plays-eval harness. Full 4-model review round applied (25 findings, 0 invalid — consensus fixes incl. the unpersisted-reservation CRITICAL, hidden AI-SDK retries, /ES↔Eversource collision, partial-spread phantom totals).

Provider decision (user, 2026-08-19): openai-codex — extraction rides the ChatGPT-subscription OAuth maintained by pi (~/.pi/agent/auth.json, mounted read-only host→CT→container), model gpt-5.6-luna. Probed then hand-rolled CodexAnalyzer (SSE-only dialect; strict json_schema + input_image both work). Metering is notional there — the daily cap bounds volume.

Live result: the entire captured backlog (20 plays) extracted, zero failures, after three live-surfaced fixes: max_output_tokens unsupported by the codex backend (400), 400/401/403 classified unbilled (reservations dropped), phantom calendar dates nulled instead of burning attempts. Typical single-position play: ~2.5k tokens in, ~$0.001–0.002 notional; direction/confidence/screenshot-kind all plausible on spot checks.

Gate remainder: the ≥30-case hand-labeled eval at ≥80 % per-field accuracy (fixtures/plays/README.md). Screenshots are accumulating live; labeling is the manual step.

[foae]

Labeled eval round 1 (2026-08-19) — model DECIDED: gpt-5.6-sol.

Answer key: 18 cases hand-labeled by an independent read of all 21 archived screenshots (Claude reading raw images, before consulting model output — satisfies the anchoring rule). Committed under fixtures/plays/ (images gitignored pending redaction).

Head-to-head, same prompt v2, same harness (~3¢ notional/run):

field	luna	sol
ticker/side/quantity/strike	100%	100%
expiry	63.6–72.7% (unstable across runs)	100%
avg/cost/value/pnl/realized/currency	100%	100%
position_count	100%	100%
screenshot_kind	83–89%	89% (both misses are taxonomy judgment calls)

luna's expiry failures are stochastic M/DD year-resolution refusals; the codex backend rejects temperature, so no determinism lever — prompting plateaued after 3 tuning rounds. sol resolves them all. Config switched (extract_model = gpt-5.6-sol; interpret stays luna for P3 text), deployed, and the whole backlog re-extracted uniformly under sol + prompt v2.

Also hardened from this round: year plausibility clamp (a model emitted 8170-08-17 — a 'real' calendar date), non-ISO currency sanitization ('USDC' on a Hyperliquid perp killed a parse), eval per-case error boundary + per-case mismatch detail, derived cost-basis convention (P5's marking needs it).

Gate remainder: grow the labeled set to ≥30 (capture accumulates ~15/day; label the deltas the same way) and re-confirm sol ≥80%/near-perfect officially. The 80% threshold is already exceeded on every field at n=18.

[foae]

Labeled eval round 2+3 (2026-08-20) — n=36, all fields ≥80%: eval gate MET on gpt-5.6-sol.

Set grown 18→36 (004848c): 18 new cases hand-labeled from the overnight capture (~126 plays accumulated), stratified per fixtures/plays/README.md — new brokers (IG, Trading 212, Questrade, Wealthsimple, E*TRADE, IBKR incl. German UI, Hyperliquid perp), multi-leg books (10-leg Webull CAD/USD mix, 11-leg IBKR covered-call tree), realized views, short legs, fractional shares, a zero-position chart case. Anchoring rule respected (labels from an independent read of the raw images).

Three n=36 runs under prompt v2 exposed two scoring artifacts (fixed in f6d0ccb: currency null≡USD normalization; monthly-expiry key correction) and one real instability: current_value flapped 70.3–86.7% across runs — the field had no stated convention, so the model alternated between copying, deriving, and nulling. Prompt v3 (c705a75) pins it (open → copy/derive from displayed mark; closed → proceeds), pins realized-P&L lists as position data, and fixes a v2 line that told the model to strip the leading futures slash the schema preserves.

Round 3 (prompt v3, n=36, ~$1.6 notional/run):

field	accuracy
ticker / side / quantity	94.6% (70/74)
strike / expiry	93.2% (69/74)
instrument / pnl_pct / realized	94.6%
current_value	91.9% (was 70–87% under v2)
avg_price / cost_basis	85.1% / 83.8%
currency / opened_at / pnl_abs	93.2% / 91.9% / 91.9%
screenshot_kind / position_count	86.1% / 97.2%

Residual misses are dominated by two hard cases: the 4-vs-5-leg straddle-splitting disagreement on a Webull realized list (defensible either way) and mixed-currency single-position screens (GBP account showing USD prices). One true model failure mode remains: occasional year-resolution slips (2024 for a 2026 post) — rare under sol (1 leg in 74).

Also this round, from live data: 6c8d85a — overlong notes (>500 chars) burned a play with 4 attempts (~$0.87 notional in retained reservations); the pre-parse sanitizer now truncates notes/broker instead of failing. Proven live: the failed play re-extracted first try after redeploy.

Ops incident (2026-08-20): pi and ~/.pi vanished from the host overnight — the worker's codex OAuth mount broke silently (the running container coasted on the stale mount; the next compose up failed). Reconstructed auth.json from the access JWT found in deploy/v2/.env — valid until 2026-08-25 ~19:39 UTC, no refresh token. @foae: re-establish the OAuth (pi → /login) before then or extraction stops (radar unaffected; rows rest at media_ready). Note the incus single-dir-mount staleness trap: after recreating the host dir the pi-auth device had to be removed and re-added.

Worker redeployed with v3; live queue extracting. Gate remainder: owner spot-check of the hand-judged sample — the machine-scored thresholds are met.

[foae]

Round 4 (prompt v4 + corrected eval contract, n=36) — the postedAt fix was the story.

A 4-model review (gpt-5.6-sol, GLM-5.2, Kimi-K3, DeepSeek-V4-Flash; 22 valid findings, scorecards filed) over this session's commits found, by 3-model consensus, that the eval never sent postedAt while production always does — the prompt's expiry-year rule ("resolve the year from the post date given below") had nothing to anchor on, so every prior round measured expiry on a contract production never uses. Fixed in 790f0f1 along with: pre-dispatch auth failures now classified unbilled (a dead refresh token was set to burn ~$0.22/attempt of daily budget after token expiry), prompt v4 (closed-SHORT current_value = debit to close; avg_price = entry, never a closing fill), CodexAuth/codex-login hardening, an offline CI lint for all 36 answer keys, and two key corrections (closing-fill avg_price → null).

field	round 3 (v3, no postedAt)	round 4 (v4 + postedAt)
ticker / side / strike / expiry	93–95%	100%
quantity	94.6%	97.3%
instrument / pnl_pct / realized / opened_at	92–95%	100%
avg_price / current_value / pnl_abs	85–92%	97.3%
cost_basis / currency	84–93%	93.2 / 94.6%
screenshot_kind / position_count	91.7 / 97.2%	91.7 / 100%

Deployed: worker rebuilt on 790f0f1 (prompt v4 + unbilled-auth classification live). All pushed through 790f0f1.

Auth: pi is retired for good (owner, 2026-08-20). New: pnpm -C packages/worker codex-login — the Codex CLI's own PKCE flow, writes the auth file, after which the worker/eval self-refresh AND persist (host-side). One interactive run needed before 2026-08-25 ~19:39 UTC (headless: ssh -L 1455:localhost:1455).

Gate: machine thresholds now met with margin on every field; owner spot-check of the labeled sample remains the human step.