pipeline todo

[macanderson]

One place for every candid "honestly, this doesn't work yet / this is on the roadmap" admission currently living in comments and docs, plus recommendations for bench-ready model configs and the change list aimed at beating Claude Code + Fable 5 on Terminal-Bench 2.1. Sources: a full-repo comment sweep (2026-08-03), ROADMAP.md, bench/READINESS.md, and the new docs/pipeline-journey.md (#1209).

Context for the recommendations: the Terminal-Bench 2.1 leaderboard's top row is Claude Code + Fable 5 at 83.8% ± 1.2% ($552.67). Stella's best published number is 58/89 = 65.2% (GLM-5.2 head-to-head, single attempt, vs Claude Code's 49.4% on the same model) — a harness win on equal models, but 18+ points below the frontier row.

---

1. Verification pipeline — stated-not-hidden gaps

• Diff-coverage overlap (ROADMAP.md §1): check the witness actually executes changed lines where coverage tooling exists — "a flip whose test never touches the diff is a coincidence, not evidence." Unbuilt; needs a coverage-tooling decision.
• Impacted-test selection as ladder evidence (ROADMAP.md §3): tool-level support shipped (P2: Implement the pure-Rust use→file resolver (Option B2 of the semantic-resolution evaluation) #443/Graph-driven impacted-test selection for Rust — resolve the run_tests scope=impacted full-suite fallback #862) but "what remains is using it as ladder evidence… a restructuring of when the baseline is observed, not a bolt-on — design first, then build."
• Judge calibration: reverts are not ground truth yet (ROADMAP.md §4, stella-pipeline/src/replay.rs): "reconciliation is stream-local; passes after a session's last terminal CI verdict stay unreconciled, and reverts are not yet a ground-truth source." Threshold auto-tuning waits on real measured data.
• OOM-kill is the documented residual ambiguity in typed test outcomes (ROADMAP.md §2).
• Judge-pass-stands-alone re-request is built but deliberately unwired (stella-pipeline/src/pipeline.rs, judge arm): "sending back was built and measured… this condition holds on MOST Terminal-Bench turns, so the request costs a turn nearly everywhere rather than on the bad ones." Revisit once diff evidence exists on TB (see §5 item 4).
• Airlock low-grain guard is a no-op today (stella-pipeline/src/witness/airlock.rs): "grain never fell below Reproduction here — the guard is a no-op today," kept for future reproduction-candidate changes.
• Reference-engine golden adapter missing (stella-pipeline/src/pipeline/tests/golden.rs, docs/replay-golden-trajectories.md): "P2: Build the reference-engine adapter and settle fixture provenance, then record reference trajectories #462 asks for reference trajectories recorded from the TS engine… It does not emit this protocol, and that — not access — is the blocker."

2. Bench / Terminal-Bench readiness

• Claim analyzer's posture copy drifted three revisions behind the adapter — every current trial would fail its posture gate (fixed in docs+bench: pipeline-journey doc, fix stale posture copies, correct witness stage ordering #1209, now pinned by test_posture_matches_the_adapters).
• stella run never exits: the turn completes, emits its terminal event, then hangs until killed #960 — Stella completes its turn but the process never exits (bench/READINESS.md §8.2): the sentinel earned reward 1.0 yet consumed its entire 300s budget and ended AgentTimeoutError; terminal complete is emitted twice. Consequences: the audited run's readiness gate is unpassable, and every trial burns its full wall-clock timeout. This is the single highest-priority bench bug.
• Budget preflight would refuse the primary (READINESS.md §6): "$76.74 actually spendable… the plan totals $86.02" — top up or re-plan before the audited run.
• Every published number is the witness-off control arm (READINESS.md §9): "each trial logged continuing without an authored witness… a lower bound on the full ladder, not a measurement of it."
• CI governance: make harbor_adapter + analyzer pytest a required context on main (READINESS.md §8.1: "The gate fired. The PR merged anyway… It is decoration.").
• loop-bench is not in CI (Nightly loop-bench regression gate in CI — catch core-loop regressions between releases #873) and stella dataset export does not exist (stella dataset export — curate redacted training trajectories from receipts (#836 first slice) #872) (website/content/docs/self-improvement.mdx).
• loop-bench harness blind spots (bench/loop-bench/README.md): requested tasks not reconciled against reported rows; multi-step datasets unsupported; a crash mid-work reports as ran (unsolved).
• Bedrock is unsupported in secure benchmarking (stella_harbor/__init__.py: "does not yet support Bedrock's multi-value credential chain").
• Frontier-Bench lane: run blocked upstream on Add leaderboard package and PR-submit CI for Harbor Hub harbor-framework/frontier-bench#1405; plan and costs in bench/evidence/frontier/SUBMISSION.md.

3. Engine / API surface (the stella-parity declared-gap matrix)

The biggest single admission in the tree (stella-parity/src/lib.rs, docs/design/engine-embedding.md §4 "Gap register"):

• pipeline.verified_run is unreachable from the API — "serve does not link stella-pipeline, so the product's defining verification ladder — and the approval gate with it — is structurally unreachable from the API."
• Goal loop and sub-agents are CLI-only — "judged multi-round runs — a defining capability for agent-app hosts — cannot be requested over the wire at all."
• Engine tuning unreachable over the wire — "of EngineConfig's ~15 knobs only max_steps is settable… an embedding host cannot pin effort, reasoning, output caps, timeouts."
• The versioned Checkpoint had zero production writers (partially addressed by feat(durability): give the checkpoint a reader, the turn refs a writer, and the object store a sweeper #1201/feat(serve): a checkpoint store behind serve, so a served turn outlives the process #1203's serve checkpoint store — re-verify the parity rows).
• Hook bus never attached in serve; calibration.drift unreachable over API; two capabilities ship ShippedUnwitnessed (UNWITNESSED_BASELINE: usize = 2).
• stella-engine embedder footguns stated in docs: hard-drop cancellation loses uncheckpointed work; the turn future is !Send.

4. Everything else the comments admit (curated)

Context plane, providers, tools, CLI, store (click to expand)

• stella-tools/src/sandbox.rs — security-relevant: "only the bash tool spawns through here. Every other path to a subprocess runs UNSANDBOXED even when this is set."
• stella-context/src/retrieval.rs — ContextQuery kinds/representation fields not honored: "both are honest gaps, not intentional policy."
• stella-context/src/embed.rs — the product-default local ONNX embedder (risk R14) is "the tracked follow-up"; hash-based embedder ships today.
• stella-model/src/vertex.rs / credential.rs — full ADC chain deferred; Windows credential write is "a placeholder for the port, not a shipped path."
• stella-model/src/openai.rs — picker collapses xhigh/max to high ("would require per-model capability gating the picker vocabulary does not yet carry").
• stella-model/src/cache_economics.rs — deck duplicates the diagnosis constants; "nothing cross-checks the two."
• stella-media — --n multi-candidate flag accepted but adapters return one image ("a caller passing n > 1 approves an n× charge and then receives one image billed 1×"); terminal preview module has no caller; media base_url unreachable from config.
• stella-cli/src/settings/merge.rs — tool-scope editor "does NOT yet honour stella.toml"; [mcp.servers] in TOML "parsed but NOT yet consumed."
• stella-cli/src/usage_cmd.rs — cloud registration is a stub; OAuth login "lands later."
• stella-cli/src/memory/observations.rs — anti-poisoning threshold under-counts ("a turn is not a task… nothing populates it yet"); memory.rs domain retagging on re-init deliberately deferred.
• stella-store/src/reconstruct.rs — stella inspect doesn't cover budget-abort synthetic results or Attachment blocks; content_free.rs — telemetry pseudonym "is dictionary-attackable by a determined intake operator"; JSONL drain is a DrainFormatGuard::NotYetBuilt declared gap (ENTERPRISE: P2: Epic: enterprise cloud telemetry — org-scoped sync, auth, retention #403/P2: Cloud drain transport: HTTPS CloudIntake adapter, drain config, and the invocation site #404).
• stella-protocol/src/context_event.rs — "schema landed, channel not yet wired. Nothing constructs or reads a LifecycleEventEnvelope today."
• stella-core self-improvement slices: Tool Foundry "proposes and installs nothing" (Self-improvement: Tool Foundry — stella authors, tests, and installs its own tools at runtime #830 first slice); self-tuning is one knob (Self-improvement: eval-driven self-tuning — a bandit over stella's own prompts, policies, and routing #831); "Nothing in Self-improvement: Tool Foundry — stella authors, tests, and installs its own tools at runtime #830, Self-improvement: stella-maintains-stella — autonomous audit → gated self-PR → merge loop #832, Self-improvement: causal self-model — predict and preempt stella's own failure modes #834, Self-improvement: self-directed capability curriculum — stella drills what it can't yet do #835, or Self-improvement (frontier): weight-space adapters — stella distills its own trajectories into evaluated LoRAs #836 is implemented" (self-improvement.mdx).
• docs/design/adaptive-context.md §4 — nine-row gap register; retrieval settings block "shipped but inert… neither has a runtime consumer"; sweep "does not yet retire anything on its own."
• docs/design/exploration-sharing.md — "the context plane cannot see explorations"; discovery pull-only; staleness repo-global and binary.
• docs/adr/0013 — "Nothing in this ADR is implemented" (session artifacts); docs/design/context-frame-spec.md §15.1 — seven-row "required before production" table.
• Serve docs vs reality: agent-engine-in-your-app.mdx "What is not built yet" (no stream resumption, no server-side conversation state, no approval routes, no /metrics) — re-verify against feat(durability): give the checkpoint a reader, the turn refs a writer, and the object store a sweeper #1201/feat(serve): a checkpoint store behind serve, so a served turn outlives the process #1203 which may have moved some of these.
• TUI accessibility (audit The TUI never positions the terminal cursor — screen readers and CJK input have nothing to track #935): "never positions the terminal cursor, which locks out screen readers and CJK input entirely"; ~2,400 lines of unreachable REPL (~2,400 lines of single-session REPL that no product path reaches #936).

---

5. Recommendations — model configs likely to bench well

All three slot into the knobs that exist today (agent_engine_config, and for bench: posture.py + STELLA_WITNESS_AUTHOR_MODEL / STELLA_TRIAGE_MODEL / STELLA_WORKER_EFFORT from #1205).

A. Frontier arm (the one that can plausibly clear 83.8%):

{
  "agent_engine_config": {
    "default_model": "anthropic/claude-fable-5",
    "pipeline_judge_model": "anthropic/claude-opus-5",   // witness author + judge: same provider, different model → witness arm ON
    "pipeline_triage_model": "anthropic/claude-haiku-4-5",
    "allowed_models": ["anthropic/claude-fable-5", "anthropic/claude-opus-5", "anthropic/claude-haiku-4-5"],
    "agents": {
      "default": { "effort": "xhigh", "reasoning": "on", "params": { "max_tokens": 128000 } },
      "worker":  { "effort": "xhigh", "reasoning": "on", "params": { "max_tokens": 128000 } },
      "judge":   { "effort": "xhigh", "reasoning": "on", "params": { "max_tokens": 64000 } },
      "triage":  { "effort": "low", "reasoning": "off" }
    }
  }
}

Rationale: xhigh, not max — Anthropic documents xhigh as the setting for coding/agentic work with max prone to overthinking, and posture.py's own comparator-parity rule says the same. max_tokens at the model's own ceiling (128K for Fable 5/Opus 5) per the "never be the side that stops first" rule — which means model_timeout (816s, tuned for 64K GLM steps) must scale with it; Fable 5 turns on hard tasks legitimately run many minutes. Fable 5 requires reasoning always-on, which the adapter's reasoning: "on" posture already matches. Triage at low/off on Haiku is pure latency/cost savings — it emits a three-line classification.

B. Cost-efficient arm (defends the $-per-solve story): worker zai/glm-5.2 (the measured 65.2% arm), judge/witness author openrouter/anthropic/claude-opus-5 via the one-gateway routing #1205 just enabled (cross-vendor judge on one credential), triage deepseek/deepseek-chat at low/off, worker xhigh/64000. Cross-family judging is the router's stated preference (L-M8) and the head-to-head's false-pass data says the judge is the weakest link — spend the premium there, not on triage.

C. Audited-claim arm: exactly the frozen protocol posture (GLM-5.1, xhigh/64000) + STELLA_WITNESS_AUTHOR_MODEL=openrouter/deepseek/deepseek-v4-pro for the treatment arm — the analyzer and adapter now provably agree on its hash (#1209).

6. Recommendations — changes to beat Claude Code + Fable 5 on Terminal-Bench 2.1

Ordered by expected points-per-effort. Target: > 83.8% over 89 tasks × 5 attempts on the canonical harness.

1. Fix stella run never exits: the turn completes, emits its terminal event, then hangs until killed #960 (process never exits). Every trial burns its full wall-clock timeout today and the readiness gate is unpassable. On time-tight tasks this converts passes into timeout zeros; nothing else on this list matters until it's fixed.
2. Run the frontier arm (config A) with ceilings at true parity. The 83.8% row is Fable 5. The three-ceilings rule is now wired (params.max_tokens, model_timeout, per-trial --turn-budget) — set them from the model's and Harbor's own limits rather than constants tuned for GLM (128K output ⇒ model_timeout must grow accordingly).
3. Turn the witness arm ON. Every published number ran with the authored-witness rung structurally off, and the measurement of what replaces it is damning: the judge alone agreed with the benchmark's grader 46% of the time, and its false passes cost 5 tasks outright. An independent author + the confirmation run (Verification §2 (Phase 1): confirmation run on flip — re-run the tracked command to guard against flaky-test false flips #859), fingerprints (Verification §2 (Phase 3): failure-fingerprint matching — a pass that "fixes" a different failure than observed is NoEvidence #867), and mutation audit (Verification §1 (Phase 4): mutation-check the witness — apply trivial mutations and downgrade a tautological witness that stays green #870) exist precisely to convert silent false-dones into caught revisions.
4. Give Terminal-Bench task dirs a git repo. TB images are plain directories, so on exactly this benchmark the diff probe is structurally blind (Verification ladder is blind on Terminal-Bench: judge says "file likely does not exist" while the file is in the container #973 — every empty diff reads "cannot read the tree"), candidate isolation is unavailable, the witness author has nothing to snapshot, and the mutation audit can't restore. One adapter-side git init && git add -A && git commit inside the container (before Stella starts) unlocks the diff evidence channel, the zero-diff guard, isolation, best-of-N, and the witness arm per-task — most of the verification ladder, on the benchmark it was built for. (Confirm the leaderboard's trajectory-review rules are comfortable with it; it's an in-container action like any other.)
5. Find and kill the 106 cap_hits. The GLM-5.2 head-to-head raw totals record cap_hits: 106 for Stella vs 0 for Claude Code on an uncapped run. Whatever ceiling fired 106 times is a "we stopped first" event by definition — identify it (step cap? output cap? compaction pressure?) and remove or raise it.
6. Token efficiency for the frontier arm. Stella spent 366M input tokens vs Claude Code's 57M (6.4×) in the head-to-head. At GLM prices that was a rounding error; at Fable 5's $10/MTok input it's the difference between ~$550 and ~$4,000 per run. Verify gateway prompt-cache hits actually land (the provider-parity matrix exists because OpenRouter once served Claude with zero caching), and look at per-step transcript growth and tool-result trimming before the expensive run.
7. Raise max_revisions (2 → 3–4) for the bench posture once the witness arm is on — revisions are the mechanism that converts near-misses, the budget guard still bounds spend, and distress guidance (Verification §4 (Phase 3): early distress guidance for repeated identical failures — trigger immediately on a repeated fingerprint #868) already fires on the second consecutive deterministic failure.
8. Best-of-N on unverified candidates (needs item 4's isolation): candidates: 2 retry-in-place for tasks whose single-shot ends Unverified/Failed would target exactly the tail that separates 65% from 84%. Opt-in per the cost model.
9. Then submit properly: all 89 tasks × ≥5 attempts, canonical dataset ref, public trajectories, maintainer review — preregistered like the GLM-5.1 claim, with the now-unblocked analyzer/posture parity (docs+bench: pipeline-journey doc, fix stale posture copies, correct witness stage ordering #1209) keeping the claim path honest.

---

The full sweep transcript (every file:quote pair, including the ones curated out above) is reproducible with the marker list in the sweep: TODO|FIXME|not yet|not implemented|deferred|placeholder|stub|for now|tracked follow-up|honest|declared gap — the repo has zero todo!()/FIXME in production code; everything above is a prose admission or a machine-checked declared gap, which is why this list is worth maintaining as the single index.

[macanderson]

§6 state reconciliation (2026-08-02)

Nothing in this issue has been ticked as work landed, so here is where §6 — the ordered change list — actually stands. §1–§4 are unchanged; they are a register of declared gaps, each its own project.

#	Item	State
6.1	Fix #960 (process never exits)	Landed — #1212 bounds runtime shutdown on the success path; #1213 added the SIGKILL post-mortem
6.2	Run the frontier arm at true ceiling parity	Blocked on a decision. #1229 landed PROPOSAL-fable-128k-posture.md as a preregistration memo with sign-off checkboxes and deliberately did not implement it — the ceiling change moves the frozen digest and model_timeout is a SUT re-freeze
6.3	Turn the witness arm ON	Mechanism ready, run outstanding. STELLA_WITNESS_AUTHOR_MODEL has existed since #1007 and §6.4 removed the last structural blocker. What is left is a scored run, not code
6.4	Give TB task dirs a git baseline	Landed — #1225 (STELLA_TB_GIT_BASELINE, default on)
6.5	Find and kill the 106 cap_hits	Landed — #1227. The premise did not survive contact: the count was inferred ("≥16384 output tokens and no tool call"), and 106-vs-0 was never a comparison. #1229's post-mortem then found 74 of the 106 were absorbed by passing trials
6.6	Token efficiency for the frontier arm	Scoped, unstarted. #1229 decomposed the 6.40× input gap as 1.58× more steps × 4.06× more input per step, and puts a 3.1× reduction on the table from recall/context engineering alone
6.7	Raise max_revisions for the bench posture	PR #1241
6.8	Best-of-N on unverified candidates	PR #1241
6.9	Submit properly (89 × ≥5, public trajectories)	Downstream of 6.2/6.3

On 6.7 and 6.8

They shared one blocker, so they are one PR. The finding worth recording here: both knobs were implemented and reachable from nothing. Outside stella-pipeline's own tests, max_revisions had exactly one writer — PipelineConfig::default(), to 2 — and candidates exactly one, to None. Best-of-N was not disabled by policy or hidden behind a flag; it had never run in production. A doc comment on candidates_run described "a --candidates 4 run", a flag that was never built.

The blocker was settings::ENGINE_ROOT_FIELDS, which the trusted-launcher gate shares and which fails closed — a posture naming an unknown root key does not lose the key, it loses the run. So exposing the knobs and letting the bench select them is one edit.

#1241 adds pipeline_max_revisions / pipeline_candidates to agent_engine_config (both encodings, all four drivers) and host-side STELLA_MAX_REVISIONS / STELLA_CANDIDATES selectors in the shape #1205 established. Unset omits the key rather than writing the default into it, so every recorded digest still reproduces byte-for-byte — anthropic/claude-sonnet-5 still hashes to 3c428a22…. No frozen constant or registered digest changes.

Two main-red fixes carried in the same PR

Both blocked the pre-push gate and predate this branch:

• verify::tests::both_judge_prompts_mark_the_diff_as_worker_authored_data asserted a diff-heading spelling that fix(pipeline): close four judge-path gaps found in the witness-protocol review #1206/Worker-agent lifecycle: judge-context sealing, single recall, persona parity, fleet timeout/pause, budgeted prompt sections #1214 reworded — it was asserting the framing was present while naming a string that no longer existed.
• stella-protocol/src/event.rs linked [stella_protocol::completion::FinishReason], naming the crate by its external name from inside itself, which is a hard error under make doc-warnings. It hid because rustdoc bails at the first failing crate, so a plain cargo doc reports one warning and never documents the rest of the workspace.