ToolDAG transfer run (Phase 3)

[pixiiidust]

What to build

Phase 3: test whether the ESL metric transfers from StatefulPuzzle to typed tool-use workflows.

Run the same runner and analyzer on the upstream tool_dag environment (SC 5/10/20/40 × DD 1/2/4/6, T=40, 10 episodes/cell, same model, new config file). Deliver ToolDAG heatmaps, ESL contour overlays, bucket tables, a comparison against the Phase 1 StatefulPuzzle boundary, and a verdict on whether ESL = SC × DD appears environment-general. Add the variable_binding_error metric only if the existing metrics demonstrably fail to expose typed workflow failure.

Reference: docs/plan.md §12; CONTEXT.md decision 6.

Acceptance criteria

• ToolDAG grid completes under both caps with its own committed config
• Same analyzer outputs generated for ToolDAG (label + realized ESL)
• Transfer comparison vs the Phase 1 boundary included
• Explicit verdict on environment-generality of ESL = SC × DD
• No new metrics added unless existing ones demonstrably fail

Blocked by

• Phase 2 ESL verdict + Step 5 decision gate #7 (gated on the Step 5 decision)

[pixiiidust]

Pre-gate design review (code-verified): upstream tool_dag is a weak dependency-chain test as implemented.

Verified against src/environments/tool_dag.py (vendored at 66c12c3):

1. Initial vars cycle TYPE_POOL[i % 8] (L83) → at SC≥8 all 8 types exist at reset; only SC=5 has type scarcity.
2. call validity checks type presence (avail_types = set(self._vars.values()), L208-211), never variable identity — and call(tool_n) takes no variable arguments at all (L125), so binding errors are unexpressible.
3. TARGET auto-binds on any call whose output type matches (L320-322).
4. Consequence: at SC≥8 zero tools are ever blocked; dag_depth is decorative; optimal play is 2 steps (call any tool with target output type → finish). DD changes what's listed per tool, not what's hard.

So: success/action-validity there would measure type lookup, not tool reasoning, and a flat surface would not disprove collapse (weak stress knob). WSA drift under a 40+ variable table is the only load-bearing signal it retains.

Proposed amendment (decision at the #7 gate): minimal 'ToolDAG-B' binding variant reusing all upstream machinery — (a) call(tool, v_i, v_j, ...) with named vars whose types must match, (b) inputs consumed/expiring so type-twins are not interchangeable, (c) no TARGET auto-bind; finish names the produced var. Type-duplicate vars then act as SC-scaled distractors and DD becomes true binding pressure. Optional: keep one cheap as-is run as a control — if binding-ToolDAG collapses earlier at equal SC×DD, identity binding is isolated as the load-bearing difficulty.

Real-world grounding: schemas machine-check types for free; production agent failures are identity/staleness/fabrication — 'a customer_id of the right type but the wrong customer'. That is the failure mode ESL needs Phase 3 to expose.

[pixiiidust]

ToolDAG-B design spec + adapted trigger gate (follows the code-review comment above; corroborated by the paper's own Table 1, where upstream ToolDAG fails the ≥20pp DD-drop criterion — the flat curve the type-presence check predicts).

Environment (tool_dag_b, new registration; upstream tool_dag untouched as control):

1. call(tool_k, v_i, v_j, ...) — named args; validity = exists + type match + provenance match (recipe pins each input to outputs of designated parent tools).
2. No TARGET auto-bind; finish(v) verifies the named variable's full provenance tree against the gold recipe.
3. Same-type wrong-provenance distractor vars seeded at reset; count scales with SC.
4. DD = number of simultaneously correct identity bindings per call.
5. Strict mode for the gate run (provenance-wrong = invalid, clean tau_a). Taint mode (wrong provenance succeeds and silently poisons downstream) reserved as a later realism ablation.
6. WSA scores the believed variable graph (var → type → producer) vs gold — the agent-believed variable graph from the transcript digest, made literal.

Trigger gate (locked before any evaluation, adapted from the paper's rule): DD sweep at fixed SC=20, DD ∈ {1..6}, n=10/level, deepseek/deepseek-v4-flash. PASS iff (a) success monotone non-increasing (no level >5pp above its predecessor) and (c) max drop ≥ 20pp. Criterion (b) (cross-model shape) waived for the single-tier budget; optionally purchasable later (~$1, 60 eps on a second cheap slug). Full Phase 3 grid runs only on PASS; a FAIL means redesign or report, not a blind grid.

Est. cost: gate sweep $1–3; full grid $10–15; optional upstream-control column ~$2.

Implementation remains gated on the #7 decision.

[pixiiidust]

Gate sweep operational note (2026-07-04). The runner's cumulative JSON-retry-rate guard auto-stopped the DD sweep at the 30-episode checkpoint (0.389 > 0.35 config threshold). Data-quality audit before resuming:

• Fallback rate (the real corruption metric): DD1 1.1%, DD2 1.6%, DD3 4.7% — retries succeed, state stays parsed.
• DD>=3 failures are dominated by wrong_provenance + variable_binding_error (203 each at DD3), zero meaningful parse-failure contribution — the provenance-binding failure mode ToolDAG-B was built to expose.
• Interim tally: DD1 10/10, DD2 10/10, DD3 1/10 (a 90pp drop), DD4 0/2 in-flight.

Raised json_retry_rate_threshold 0.35 -> 0.6 (commit 8da82d3, maintainer-approved) and resuming — same operational adjustment Phase 1 made (0.08 -> 0.35). The locked plan §12 gate rule (monotone non-increasing + >=20pp max drop) is unchanged; verdict remains mechanical via check_gate.py on the full 60 episodes.

[pixiiidust]

GATE VERDICT: PASS (2026-07-05, check_gate.py, locked plan §12 rule)

DD	success
1	10/10 (100%)
2	10/10 (100%)
3	1/10 (10%)
4	0/10 (0%)
5	0/10 (0%)
6	0/10 (0%)

• (a) monotone non-increasing: True (max rise 0.0pp)
• (c) max drop >= 20pp: True (max drop 100.0pp, DD1→DD4)
• 60/60 unique episodes, no duplicate task_ids. Gate cost ~$3.9 (caps never hit; two operational stops documented above: retry-rate guard at ep30, output-token mean at ep59, one code-error episode retried clean).
• Failure mode is the designed one: wrong_provenance + variable_binding_error dominate; fallbacks <5%; ceiling hits 2 (at 16k updater cap).

Provenance binding is a readable, brutally sharp dependency axis at SC=20 — far sharper than StatefulPuzzle's (10/10/9/7 at the same SC). Full Phase 3 transfer grid launched per the maintainer's pre-approval: configs/tooldag_b_esl_160.json (SC {5,10,20,40} × DD {1,2,4,6}, T=40, 10/cell, caps $20/200M, updater cap 32k). Staging PR: #19.

[pixiiidust]

PHASE 3 TRANSFER GRID COMPLETE (2026-07-05, configs/tooldag_b_esl_160.json, 160/160 unique episodes, $5.53 list, no cap stops)

Final surface (success n/10):

SC \ DD	1	2	4	6
5	10	10	10	10
10	10	10	10	10
20	10	10	0	0
40	10	0	0	0

• Binary-sharp: every cell 10/10 or 0/10. Dominant axis flips vs StatefulPuzzle: DD-dominant with an SC×DD interaction (cliff at DD=4 for SC=20, DD=2 for SC=40; SC≤10 survives DD=6; SC=40 DD=1 passes — the cell class that died in Phase 1).
• Simple SC×DD is the top predictor here (Spearman 0.810 > token_count 0.804 > SC+DD 0.746), within-bucket variance 0.0000; label surface separates perfectly at ESL ≤60 pass / ≥80 fail. The same threshold is violated in StatefulPuzzle — no fixed formula transfers.
• World-state-first signal at the primary WSA<0.5 threshold: 43 paired collapses, 67% ws-first, median lag −1 (Phase 1 needed the 0.9 sensitivity threshold). WSA tracks SC while success tracks DD — SC=40 DD=1 succeeds 10/10 with WSA 0.55.
• Failure composition: variable_binding_error 1,328 + wrong_provenance 1,327 vs 1 parse error in 2,811 steps. Fallback 2.9%; 15 ceiling hits all in already-failing cells; truncation guard 0.
• Operating point (analysis/openrouter/operating_point.py): inside-boundary cost flat ($0.004–0.025/delivered episode); past-cliff episodes cost 4–6× more per attempt and deliver nothing.

Verdict (#8 acceptance item) — full analysis in docs/reports/phase3-transfer.md (commit 7df9da8): ESL = SC × DD is environment-general as a metric family + measurement recipe (sharp mappable boundary, multiplicative form ranks at/near top in both environments, genuine non-artifact failures), not as a fixed formula (boundary location and axis weights are environment-specific, opposite dominance). Per-workload calibration is the consequence.

All acceptance boxes are now satisfiable; closing decision left to the maintainer.