Observatory: provenance, rich rendering, evolution replay, descriptor atlas, live view - #197
Closed
lfnothias wants to merge 37 commits into
Closed
Observatory: provenance, rich rendering, evolution replay, descriptor atlas, live view#197lfnothias wants to merge 37 commits into
lfnothias wants to merge 37 commits into
Conversation
- factory.py: replace infinite MCP discovery loop with 5-retry exponential backoff - orchestrator.py: classify sandbox errors as TIMEOUT/SYNTAX_ERROR/RUNTIME_ERROR - workflow_factory.py: validate Python keyword identifiers, atomic state_result.json write, post-assemble compile check - workflow_runner.py: delete temp exec script in finally block - evolution_engine.py: ghost WorkflowInfo guard (generation_failed uuid); ASTRA export returns bool + .export_status sidecar; clear VariationEngine score/gradient/agent-count histories on session start - selection.py: refresh member metrics before eviction check (not after) - planner.py: verify expected outputs of completed dependencies before marking step executable - smolagent_factory.py: file lock on save_memories; explicit TimeoutError re-raise before generic handler; restore exception capture in _run_agent retry loop; max_steps=35 + context-window guard callback - astra_exporter.py: fail-fast on empty trace - llm_provider.py: strip temperature entirely for Opus 4.x (invalid_request_error); broaden temperature error detection to message string - variation_engine.py: cap variation temperature at 1.0 for Claude models - csv_mode.py: parse SUCCESS_LEVEL from LLM response; pass scenario_rubric filename; resolve runs_capsule_dir to absolute path - config.py: add max_steps, max_context_tokens, export_astra fields - workflow_v11.md: add MCP response schema guidance, absolute-path rule, env-setup via shell tool, no built-in file I/O rule Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The retry loop in LLMProvider.__call__ was `while True`, and the `self.max_retries` field (set in __init__) was never read. A persistently overloaded or rate-limited provider therefore retried indefinitely, re-sending the full prompt on every attempt — unbounded latency and, on providers that bill partial/streamed output, unbounded token spend. Both retry paths (timeout and generic retryable errors) now raise once `attempt >= max_retries` instead of looping forever. The context-window shrink path keeps its existing independent 3-step cap. Default max_retries raised 3 -> 6 so exponential backoff (capped at 500s) still rides out transient blips while guaranteeing termination. No behaviour change on the success path or for non-retryable errors. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The repo ships a tests/ suite using the `*_test.py` pytest convention, but
pytest was not declared anywhere, so `uv sync` produced an env that could not
run the tests (No module named pytest). This adds a PEP 735 dev group so the
suite is runnable with:
uv sync --dev
uv run pytest tests/
No runtime dependency change (dev group only); uv.lock is gitignored in this
repo, so it is intentionally not committed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The verifier is the dominant LLM cost: a single evaluation fires 50-100+ judge calls (claim extraction across sources, dedup, importance rating, per-claim file selection, verifier-script generation, package checks), and every one ran on judge_model. Most of those are mechanical structured-JSON extraction, not calibrated judgement — they do not need the strong/expensive judge. This adds an optional `judge_extraction_model` config field. When unset (default None) it falls back to judge_model, so behaviour is unchanged. When set, the six mechanical call sites route through it via a new `use_extraction_model` flag on `_call_judge` / `_call_judge_for_json`: - claim extraction (per source) verifier_claims._extract_claims - near-duplicate dedup verifier_claims._run_dedup_pass - importance rating (per batch) verifier_claims._rate_one_importance_batch - per-claim file selection verifier_per_claim._llm_select_files - verifier-script generation verifier_per_claim._call_and_parse_verifier - missing-package check verifier_per_claim._llm_packages_needed_for_claim The two calls that are genuine judgement stay on judge_model: the per-claim soft verdict (_score_soft) and the mutation textual-gradient (_build_abstractec_textual_gradient). Expected saving: with judge_extraction_model pointed at a cheap tier, the large majority of judge calls per evaluation (and per evolution iteration) drop from the strong model to the cheap one, while final scoring quality is preserved. Judge-LLM construction is centralised in _build_judge_llm_config so both tiers are configured identically (temperature, routing). The importance test's judge monkeypatch gains **_kwargs to match the new signature. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a `model_tiers` map ({"heavy": ..., "light": ...}) so any *_model role
(planner, workflow, smolagent, judge, judge_extraction, capsule_namer) may be
set to a tier alias instead of a concrete id. Aliases are resolved to the
tier's model at load time via Config.resolve_model, so the whole fleet's cost
profile can be switched by editing two lines, e.g.:
"model_tiers": {"heavy": "openrouter/z-ai/glm-5.2",
"light": "openrouter/minimax/minimax-m3"},
"planner_llm_model": "heavy",
"judge_model": "heavy",
"judge_extraction_model": "light",
"smolagent_model_id": "light"
Purely additive and backwards compatible: a concrete model id (or None) is
returned unchanged, and the default role values are concrete ids, so the tiers
are inert until a role opts in. This pairs with the judge_extraction_model tier
(extraction -> "light") and the OpenRouter caching/routing already in place.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Agent and verifier both grounded via Perspicacite with a hardcoded kb_name=None (web search). For a fair benchmark these must differ: the verifier may use full ground truth, but the agent must use only a leak-free tier — and unrestricted web search can surface the source paper, leaking answers even with the browser MCP disabled. This threads kb_name through query_perspicacite (and the streaming/non- streaming transports + the on-disk cache key, which now separates per-KB entries while keeping the existing kb_name=None keys valid), and adds three config fields: - perspicacite_agent_grounding_enabled (default True) — agent-side opt-out - perspicacite_agent_kb_name (default None) — KB the agent grounds from - perspicacite_verifier_kb_name(default None) — KB the verifier grounds from Agent callers (orchestrator + planner) honour the enable flag and pass the agent KB; the verifier passes its own KB to get_perspicacite_grounding. All defaults reproduce the current behaviour, so this is opt-in. It enables the grounding ablation: agent = none / leak-free brief KB, verifier = ground-truth KB. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ing) The verifier graded every extracted claim (default cap 100) with grounding always on; on a real run this expanded to ~80 per-claim gen/exec/score loops and dominated wall-clock and OpenRouter spend. Both were constructor-only and not reachable from config. Adds two optional config fields, plumbed at the VerifierEvaluator construction site: - verifier_max_claims (default None -> verifier's own default of 100) - verifier_use_grounding (default None -> verifier's own default of True) When unset they are omitted, so the verifier keeps its library defaults (zero behaviour change). Lowering verifier_max_claims grades only the top-N most important claims for a faster/cheaper run, trading coverage of low-priority claims for speed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A fair run revealed the agent could not execute a data-analysis reproduction: the smolagent sandbox authorized pandas/numpy but not sklearn or scipy (and subprocess is blocked), so it could not train a RandomForest, run PCA/PCoA, or compute distance metrics in-sandbox — it fell back without producing the deliverables. Adds scipy + scikit-learn (and common submodules) to additional_authorized_ imports in both agent factories, and scikit-learn/scipy to runner_requirements (default config + the fair eval config) so they are installed in the runner venv. This lets the agent perform ML/stats reproduction in Python without shelling out, which is the general capability such tasks need. General (any data-science reproduction benefits), not task-specific. These are standard data libraries with no system-access surface beyond what is already allowed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 17 variation_engine_test failures were stale, not real regressions:
- VariationEngine() was called with no args, but the constructor gained a
required `config` (it builds the mutation LLM config). Pass a real Config().
- Two assertions checked for old scope-label keywords ("EXPLOITATION",
"RE-SPECIATION") that the mutation prompt no longer emits; the surrounding
logic assertions (boldness, respeciation_gate_open) still pass, so only the
display strings drifted — updated to the current labels ("Slight mutation",
"Bolder mutation").
- test_mutation_prompt_does_not_touch_gradient_history made a live LLM call as
a side effect; it verifies gradient-history invariance, not LLM behaviour, so
the directive call is now stubbed (no network).
Full suite: 109 passed, 4 skipped, 0 failed (was 92 passed / 17 failed).
Test-only change; no production code touched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…fier knobs Adds unit coverage for this session's cost/fairness config features (pure config behaviour, no network/LLM): - model_tiers + resolve_model: alias -> model, concrete passthrough, None - defaults are concrete ids so tiers stay inert (zero behaviour change); judge_extraction_model defaults None (reuse judge_model) - from_json resolves tier aliases on every model role - per-side Perspicacite KB + verifier cost knobs: defaults reproduce current behaviour, and dump/load round-trips them Locks in the opt-in/zero-default-change contract these features were built to preserve. Suite: 114 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Documents the config-driven cost levers (model tiers, cheap judge-extraction tier, prompt caching, verifier_max_claims/use_grounding, bounded retry) and the leak-free fairness controls (per-side Perspicacite KB + agent grounding opt-out, leak linting, internet isolation, the sandbox scientific-import capability) added this cycle, plus the fair-run recipe. Emphasises that every knob is opt-in with behaviour-preserving defaults. Linked into the Evaluation nav. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…g is off make_scientific_grounded_prompt printed the "🔬 Querying Perspicacite-AI…" phase banner (and the retrieved-context line) before calling perspicacite_grounding(), which already returns a no-context sentinel when agent-side grounding is disabled. So a fair run with perspicacite_agent_grounding_enabled=false logged a query it never made. Guard both prints behind the enabled flag; the grounding call (and its internal gate) are unchanged. Logging-only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Regression guard for feat(sandbox): a fair run proved the agent cannot execute a data-analysis reproduction (RandomForest, PCoA) without scipy/sklearn. These tests assert both agent factories keep sklearn/scipy in the sandbox's authorized imports and that the runner requirements install scikit-learn/scipy (plus pandas/numpy), so the capability cannot be silently dropped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ervable Found while running Mimosa against ASB capsules on OpenRouter's stealth tier. Temperature. workflow_factory samples random.uniform(0.7, 1.3) for every workflow generation, clamping to 1.0 only for Anthropic. Backends that cap at 1.0 refuse the rest. The retry path already knew how to recover — drop to 1.0, retry — but _is_temperature_error only read error.param, which OpenAI-style APIs set and gateways do not: OpenRouter forwards the refusal as a bare 400 whose metadata.raw is the string "ERROR". So the recovery never fired and the task aborted instead. Measured against stealth/ox-alpha: temperature 1.0 and 0.7 succeed, 1.3 returns that 400 — roughly half of all generations died. Detection now also treats a 400 on a request above the 1.0 ceiling as a temperature rejection; a 400 at or below it is left alone so real bad-requests are not masked. Grounding. Every failure path in query_perspicacite returns None and each caller substitutes a "no relevant scientific context" string, so a run where grounding failed outright produces the same artifacts as a grounded one. Added a per-attempt ledger and grounding_stats(), folded into run_metrics.json, so a run can be shown to have been grounded rather than assumed to have been. Also: - perspicacite_kb_name / _mode / _max_papers config knobs. kb_name was hardcoded None, so retrieval always took the web-search pipeline and the local knowledge bases were unreachable. Default stays None — behaviour unchanged unless set. For benchmarks, never scope it to a KB built from the paper under reproduction; that hands the agent the graded values. - Persist reasoning_effort only when it was actually sent. It is gated to the o1/o3/gpt-5 families, but the configured value was recorded for every model, putting a parameter in the run's provenance the provider never received. - Precheck now names save_logprobs when it is on and every probe failed. The probe pairs logprobs with require_parameters, so an endpoint that simply does not advertise logprobs fails with a routing-shaped 404 and the generic advice sent operators to the provider allowlist instead. - Drop _OVERALL_TIMEOUT: defined, documented, never passed to httpx. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The flag was read in one place — orchestrator._ground_with_perspicacite. planner.py and the judge's grounding evaluator (via verifier.py and generic.py) call query_perspicacite directly, so turning it off still left two of the four call sites querying the literature. The recorded workaround was to point PERSPICACITE_API_URL at an unreachable host, which is also what the fair-eval notes in the dev hub prescribe for benchmark runs. Gating in the client instead covers every caller from one place. Skipped calls are recorded as "disabled" and excluded from the hit-rate denominator, so a deliberately ungrounded run is not reported as a run whose grounding failed. Default stays enabled; behaviour is unchanged unless the flag is set. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Observed running ASB capsules against stealth/ox-alpha: the planner failed all three attempts with "Failed to extract valid JSON from LLM response" / "Invalid control character at …" and the task was abandoned. The model had pasted a multi-line span straight into a string value. _extract_json_from_code_block called bare json.loads on the fenced block, so an otherwise complete plan was discarded over an unescaped newline. The repair ladder that already exists in onboard_cli.py never covered this path. Adds sources/utils/llm_json.loads_llm_json, ported from ASB's llm_pipeline helper, which solved the same problem against the same model family. It repairs raw control characters, bare interior quotes, and trailing prose after a complete object — but only after a strict parse fails, so valid JSON is returned byte-for-byte unchanged, and it re-raises the original JSONDecodeError when nothing parses so callers still see the true defect. The per-claim verifier and workflow_info parse LLM JSON the same way and are candidates for the same treatment; left alone here because neither was observed failing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
run_cached runs the agent on a worker thread, joins with a deadline, and treats
completed=False as a timeout. The retry loop caught every exception but the two
lines recording it were commented out:
#result['exception'] = e
#result['completed'] = True
So an agent whose three attempts all raised left completed=False and
exception=None. The thread exited, join() returned immediately, and the caller
raised
TimeoutError: Agent 'reconstructor' execution timed out after 18000 seconds
about two minutes into the run, with the real exception discarded — visible
only as a bare print() inside the worker. Every agent-level crash was
misreported as a five-hour timeout.
The discriminator is whether the worker is still alive after the join: alive
past the deadline is a genuine timeout; finished without completing means every
retry raised, so re-raise the last one. A successful attempt clears any earlier
retry's exception, since the caller re-raises whatever is left in the slot.
Observed against stealth/ox-alpha running ASB capsules, where the underlying
cause was the temperature-400 fixed in 6499f03 — invisible behind the timeout.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…nstall
Exit code. run_single_thread_eval_loop catches per-row exceptions and continues
— right for a batch — recording success_level: "Error" in execution_history.
Nothing read it, so a run whose every row failed exited 0. Observed: a task
logging "Error in csv row 1: Planner: Execution failed" twice, exit 0. Any
harness scoring by exit code counts that as a pass. papers_mode now reports the
count and exits 1.
Test isolation. Writing the test above surfaced a worse problem. pyproject
force-includes main.py and config.py as top-level modules, so installing the
project drops copies into site-packages, and under pytest those copies won
every import:
import config -> .venv/lib/python3.11/site-packages/config.py (Aug 14)
sources/ is not shipped that way and correctly resolved to the tree, which is
why only main/config were affected — and why it went unnoticed. The effect is
that config_roundtrip_test.py, whose whole purpose is to check which fields
survive dump/load, was checking an install-time snapshot: fields added in the
working tree were invisible to it and it passed regardless.
conftest.py now puts the repo root at the front of sys.path. config_roundtrip
consequently exercises real code for the first time in a while, and still
passes, including over the perspicacite_* fields added in 6499f03.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lane1_t0 (p_iimn) ran with the control-character repair already in place and
still lost its plan after three attempts:
Expecting property name enclosed in double quotes: line 47 column 64
That message reads like a quoting fault; it is a trailing comma — the one JSON
forbids and both JavaScript and Python allow. Six occurrences in that one run.
strip_trailing_commas drops a comma followed only by } or ], tracking string
state so commas inside values survive. The ladder now tries strict, then
string-repair, then comma-strip, then both — a real response can carry a pasted
multi-line span and a trailing comma at once.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_extract_json_from_code_block scanned only for ```json fences and returned None otherwise, which the caller turns into "Failed to extract valid JSON from LLM response" and, after three attempts, abandons the task. A model told to answer in JSON frequently just answers in JSON. Verified against the persisted plan_creator response from the failed lane1_t1 run: a valid 7558-character plan object, two top-level keys, three steps, no fence — discarded. Same failure on lane2_t0. That made it the most common terminal failure in the sample, ahead of anything model-quality related. Now tries the fenced block first (unchanged when present), then the bare response, then from the first brace so a leading "Here is the plan:" does not cost the plan either. Still returns None when there is genuinely no JSON. Re-running the extractor over that saved response now recovers the plan. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Same behaviour, spelled out as the three cases it actually handles: opening quote, closing quote, unescaped interior quote. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A response cut off at max_tokens arrives with finish_reason="length" and no exception, and the caller parses it as if it were complete. Reasoning models make this routine: the budget is spent on reasoning before any content is emitted, so a budget that would comfortably fit the answer still returns a fragment. Observed on the p_iimn re-run against stealth/ox-alpha, with every JSON repair already in place: the planner call returned Completion: 8192 against a max_tokens of exactly 8192, four truncation warnings in one run, and JSON that ended mid-string. The parser failed six times with "Unterminated string" and the task was abandoned. No repair recovers a document that was never finished. The budget now doubles and retries, at most twice and never past 65536. The config value is not mutated, so escalation is per-call. When it is still truncated after the last escalation the warning says so explicitly instead of suggesting the operator "consider increasing max_tokens" — by then that advice has already been taken automatically. This is the same shape as the existing context-window handler, which halves an oversized prompt and retries; that one guards the input side, this one the output side. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Once the temperature fallback landed, every temp>1.0 generation recovered: the log shows 1.20, 1.10, 1.24, 1.08 each falling back to 1.0 and then "generated in Ns". The generation failures that remained were all at temperature <= 1.0 — a different cause. That cause is OpenRouter reporting an upstream fault as HTTP 400 with message "Provider returned error" and metadata.raw "ERROR". The 400 makes it look like a malformed request. It is not: six identical calls at temperature 0.85 with a full-size workflow prompt succeeded in isolation while the same shape was failing intermittently under four concurrent lanes. Semantically it is a 502. _is_retryable_error did not match that wording, so it raised immediately with no backoff and each occurrence cost a whole workflow generation — the single largest remaining source of lost work in the sample. The match is deliberately narrow, on the gateway's own phrasing rather than on 400s in general, so a genuinely malformed request is not retried in a loop; it would keep failing and still exhaust the existing retry ceiling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The step-execution handler wrapped the cause as
raise Exception(f"❌ Critical error in step execution: {str(e)}") from e
`from e` preserves the chain for a Python caller, but the operator only ever
sees the formatted string. A real run ended with
❌ Critical error in step execution: unhashable type: 'slice'
— no file, no line, no exception type, and nothing in the log to grep for. The
run had already produced its 13.9 kB deliverable and written its ASTRA capsule,
so the bare message made a late failure look like a total loss.
Now logs the traceback and names the exception type, keeping `from e`.
Planner had no logger at all — `self.logger` appeared nowhere in the file — so
logging the traceback needed one added to __init__ first. Without that the fix
would have raised AttributeError from inside an except block, which is a worse
failure than the one it was meant to explain.
The underlying TypeError is not fixed here: it is a dict indexed with a slice
somewhere under run_attempts, and it is not locatable from the message alone.
That is the point — this change is what makes the next occurrence diagnosable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The p_iimn run of 2026-08-22 (20260822_183340_88cdd1fe) scored 0.760,
wrote reproduction_iimn.md and exported its ASTRA capsule, then died with
Critical error in step execution: unhashable type: 'slice'
and was reported as a 0% success rate with a non-zero exit.
The cause was the TTS summary at the end of run_attempts, which sliced
each element of final_answers. Agents do not answer with strings: every
entry of that run's state_result.json is a dict, so x[:128] indexed a
dict with a slice. Reproduced from the persisted answers on the project
interpreter. On 3.12+, where slices became hashable, the same line
degrades to a KeyError instead — the fix covers both.
Every other consumer already coerced first (planner.py ~397,
evolution_engine.py ~245); this one did not, and the annotations said
list[str] while the data said list[dict]. Corrected both.
The second defect is that the narration sat in the step's success path
at all. A line that is spoken aloud must never decide whether the work
counts, so it moves to _narrate_step_completion and is caught there —
logged with its traceback, not swallowed.
Also anchors the exception-chaining test on the raise rather than on the
message text, which now appears in prose earlier in the module.
The p_iimn re-run of 2026-08-22 reached step 4 of 6 and died with
❌ Planner: Execution failed: EOF when reading a line
Two defects, one behind the other.
request_user_exit called input() on the benchmark path. In a batch run
stdout is redirected and stdin is not a TTY, so the prompt printed into a
log and the read raised EOFError — a message naming neither the question
nor the step. It now raises UserInterventionRequired carrying the
question. Raising rather than exit(1) is deliberate: the CSV harness
counts the row and still prints its summary, which SystemExit from inside
the planner would skip. pricing.py and csv_mode._prompt_with_default were
already headless-safe; this was the last blocking prompt.
It should never have been asked. data_acquisition declared
/workspace/data/ as an output and wrote nine files into it, but
_verify_expected_outputs compares against a scan that yields files only,
so a directory could never match. The step was permanently "missing
outputs" and blocked every step downstream. A declared directory is now
satisfied by any file inside it, matched on the trailing directory name
because plans declare workspace-absolute paths while the scan returns
relative ones.
15 tests, including that an interactive yes still continues and an
interactive no still exits. Suite: 346 passed, the same 16 pre-existing
failures.
The rubric cache exists so "verifier scores stay comparable across iterations
of the same task" (verifier_claim_cache_test.py) — the first run freezes the
ranked claim list and later runs reuse it verbatim (VerifierEvaluator.evaluate).
It keys on sha256(goal). But the goal the verifier reads is the
knowledge-wrapped one, and the retry loop prepends the previous attempt's whole
answer dict to it before re-running the same step, so every retry hashes a
longer string, misses the cache, and writes a fresh rubric.
Measured on the run this came from (p_iimn task_001, step
task_and_resource_discovery, three attempts):
original_task_<uuid>.txt 2657 B, md5 02b411d9… — identical all three
goal_<uuid>.txt 2657 → 6505 → 11353 B — grows every attempt
key(goal) a19ee032 / d11aa80f / 3530d233 — 3 rubrics
key(original) a19ee032 / a19ee032 / a19ee032 — 1 rubric
The three attempts drew 21, 24 and 20 claims with disjoint ids and scored
0.000 / 0.649 / 0.557 — three numbers on three different scales, which
evolution_engine then compares with max() and reports as "Best run".
_extract_claims now takes cache_key_text separately from goal, so the claim
writer still sees the full wrapped goal (prior-attempt context is useful in the
prompt) while the cache keys on the invariant task. WorkflowInfo already keeps
that unwrapped task "for similarity matching", and record_lineage and the
variation prompts already prefer `original_task or goal`; the verifier's key was
the one place that did not. Default is falsy, so every other caller is unchanged.
The 16 failing tests in the suite (evaluation_cli, failure_fingerprint, pricing,
logprobs, csv_mode_logging) fail identically at 67b4251 with these two files
reverted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Issue #196. Step reproduction_spec_analysis declared workspace/analysis/iimn_reproduction_plan.md, wrote iimn_reproduction_guide.md at the workspace root instead, was scored 0.799 and accepted, and then killed the run at the next step's dependency gate. None of the 26 claims in its frozen rubric mentioned the declared path — claims are extracted partly from execution text and workspace listing, so the rubric inherited the agent's choice of deliverable and could not catch the agent choosing wrong. expected_outputs is the anchor: the planner writes it before the step runs, so it cannot have been shaped by execution. The verifier never saw it — it is handed a uuid, and neither state_result.json nor the workflow folder carries the declaration. Threading it through start_workflow_evolution, the generation loop, IndividualRun and the factory would touch five shared signatures. It is not needed: the planner already passes original_task=step_task, and that is the same string the verifier keys its rubric cache on. So the declaration is written under that key and read back under it — one write in the planner, one read in the verifier, no signature changed. The claims are prepended, so claims[:max_claims] cannot drop them, and added after _extract_claims has persisted the rubric, so they never enter the frozen cache: the cache freezes what a run produced, these belong to the plan and must follow it when it changes. Every failure path degrades to the previous behaviour — no declaration, unreadable file, or an exploding wf_info costs the extra claims, never the evaluation. One design question is deliberately left open in #196 rather than settled here: this makes the score partly a function of the planner's path conventions, so a step that produces an equivalent artefact under another name now fails a maximum-importance claim. That is the intended reading of "declared output", but it is Martin's call. 17 tests, including the one that pins planner and verifier to the same key function — if those drift the declaration is written where nothing reads it, and a silent stop is the failure mode this exists to end. Suite: 369 pass; the same 16 failures are pre-existing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Issue #196. Both inner branches of the acceptance block assigned TaskStatus.COMPLETED and broke, differing only in print_ok against print_warn: outputs_produced, missing_outputs = self._verify_expected_outputs(step) step.status = TaskStatus.COMPLETED if outputs_produced: print_ok(...); break else: print_warn(...); break So _verify_expected_outputs computed the right answer and nothing acted on it. Observed on a real run: step reproduction_spec_analysis was accepted at 0.799 without writing workspace/analysis/iimn_reproduction_plan.md, and the run died at the *next* step's dependency gate — with a third attempt still unspent. Now: while attempts remain, spend one on producing the deliverable rather than banking the miss. On the last attempt keep COMPLETED, deliberately, so the dependency gate still reports which output is missing for which step instead of a generic step failure replacing that precise message. PlanStep.missing_outputs carries the miss so COMPLETED alone no longer implies the deliverable exists. Checked the tests first, per the standing rule: headless_planner_test covers _verify_expected_outputs and planner_test covers _can_execute_step, but nothing asserted what run_attempts does with the result. Uncovered, not deliberate. Interacts with 8be2b35, as noted on the issue: mandatory expected_outputs claims make a name mismatch louder, this makes it consequential. A step that produced the right content under the wrong name now retries. Whether that is the desired reading of expected_outputs is Martin's call and is flagged there, not assumed here. 9 tests. Suite: 378 pass; the same 16 failures are pre-existing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Caught in production on run 6. Step `data_acquisition` declared two outputs:
workspace/p_iimn_task_001/data/
workspace/p_iimn_task_001/data/dataset_inventory.csv
as_claims worded both identically — "A file exists at that path, in the
workspace, and is non-empty" — so the directory became a maximum-importance
claim no correct run could satisfy. A step that did exactly what the plan asked
would have been marked down for it, which is the opposite of what 8be2b35 is
for.
Planner._verify_expected_outputs already makes this distinction (it short-
circuits a trailing-separator output through _directory_output_satisfied). The
claim now makes it too: a directory output asks for a directory holding at
least one non-empty file.
The existing test only asserted the claim's id, which was correct for both
kinds and so said nothing about the description. Four tests now cover the
distinction, including the exact pair run 6 recorded.
Suite: 382 pass; the same 16 failures are pre-existing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Brings 123 commits of mimosa_v2 into the personal branch, which had not been
synced since June. Ten files conflicted. Resolution rule: where Martin made an
architectural call or solved the same problem, his choice stands; only genuinely
additive, generalizable work is carried over from this branch.
Martin's, taken whole:
llm_provider.py prompt caching (_supports_prompt_caching/_apply_cache_control)
and _is_claude_model, which generalises this branch's
Opus-4-only _is_no_temperature_model — "omit temperature
for all Anthropic models rather than version-gate"
factory.py deadline-bounded MCP discovery (300 s, capped backoff)
supersedes the fixed 5-attempt loop
csv_mode.py sab_files_transfer is async with two await call sites;
_format_goal_mode_results, _format_task_mode_results,
_analyze_results and _analyze_results_isolated were
deliberately removed and are not resurrected
smolagent_factory.py model=None, temperature=1.0, max_steps=128
evolution_engine.py rewards_history guard
config.py model defaults (deepseek-v4-pro/flash), export_astra,
vision_judge_model
planner.py _verify_expected_outputs(dep_task) — this branch passed
a list where a PlanStep is expected, a latent bug
pyproject.toml dependency-groups pytest>=9.1.1; this branch's identical
pytest>=8.0 group was a duplicate and is dropped, its
comment kept
Carried over, because mimosa_v2 has no equivalent:
config.py judge_extraction_model, model_tiers + resolve_model,
per-side Perspicacite KB selection, verifier cost knobs
— all default to None/inert, so behaviour is unchanged
until set, and all are persisted so the config-roundtrip
guard stays green
planner.py don't announce "Querying Perspicacite" when agent-side
grounding is off; the call returns a no-context sentinel
workflow_factory.py MAX_CONTEXT_TOKENS/WORKSPACE_DIR alongside SAVE_LOGPROBS
variation_engine_test stubs llm_think_mutation_directive so the gradient-
history test stops requiring OPENAI_API_KEY, keeping the
assertion identical — decided by running both versions
One defect introduced and fixed during the merge: dropping
_is_no_temperature_model left a live call at llm_provider.py:464, which the
variation-engine suite caught. It now calls _is_claude_model.
Suite on the merged tree: 16 failed, 225 passed.
Plain mimosa_v2: 17 failed, 217 passed.
No failure is added; one is fixed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Second half of the sync. The first merge used the stale local mimosa_v2 (0c35d67); this brings the 14 commits that were only on origin, including the two that bear directly on the earlier resolution: 64f9ceb fix(config): restore fields dropped by the JSON round-trip 3848248 Merge PR #187 propose/config-roundtrip-coverage 17c7d30 test(planner): cover dependency-task expected-output verification c4747da fix(pricing): fall back to default pricing instead of prompting Two conflicts, both resolved to Martin: llm_provider.py max_retries stays 100. This branch's bdba070 capped it at 6 on the rationale that "the retry loop was `while True` and this value was never read" — no longer true: upstream reads it at both retry sites, so the cap is Martin's and only the value differed. config.py his discovery_addresses round-trip fix (absent key keeps the default rather than falling back to [], which left MCP discovery with no address range). The model_tiers load line is kept alongside it. tests/config_roundtrip_test.py now runs against this branch and passes, so every field carried over from the personal branch is properly persisted. Suite: 16 failed, 237 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Confirmed against this fix in production, and it went the wrong way.
Run 6 step `feature_detection_and_processing` declared five outputs. No raw
LC-MS data was reachable, so the run wrote each file as a labelled placeholder
— line 2 of feature_table_qtof.csv reads
# PLACEHOLDER: MZmine Processing Output for qTOF (MSV000080492)
# Status: NO REAL DATA AVAILABLE
— and all five mandatory claims PASSED at importance 10, one on the detail line
"File exists and is non-empty (1462 bytes)". So 8be2b35 contributed 50
importance-points of passes for files that state in their own second line that
they contain no data.
The wording was the defect. "A file exists at that path and is non-empty" is
satisfied by a stub, which means the claim pressures an agent to create the file
without pressuring it to fill the file. Under five such claims a run that cannot
obtain its inputs is pushed toward writing something — here honest placeholders,
but fabrication would have satisfied the claim equally well, and this fix would
have been the cause.
The claim now asks for content and names the failure mode: not a placeholder,
stub, template, or a note recording that the data could not be obtained.
The step still failed overall (0.552, hard_fail_capped), so other claims caught
what these missed — but these were on the wrong side of it.
Suite: 384 pass; the same 16 failures are pre-existing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Third defect in this fix, and the first one that punished honesty. 696bae2 disqualified "a file whose body announces missing or unavailable data". An A/B probe against run 6's archived workspace applied that to processing_log.md — 292 lines, per-dataset MZmine parameters, a parameter comparison table and a recovery path, the most substantive artefact in that workspace — and failed it, because the log honestly recorded that its *sibling* outputs were placeholders. The wording could not tell a file that IS a stub from a file that REPORTS on one, so it penalised exactly the behaviour the mechanism exists to protect. Now split by suffix, the same discriminator asb_eval already uses: .csv .tsv .mgf .json .parquet .mzml .mztab "holds actual records — data rows, spectra, or entries — and not merely comments, headers, or placeholder text standing in for data" everything else "substantive content produced by this step... A report that documents what was attempted and honestly records missing inputs or limitations does satisfy this claim" Verified against the shipped wording on run 6's real artefacts: stub .mgf fail "only placeholder text; no actual spectral records" stub .tsv fail "only header and/or comments, no actual records" stub .mgf fail "contains placeholder markers" real .md pass "substantive (size > 1KB, non-empty lines)" and previously, on the .csv that started this: old wording pass, new wording fail ("could not be parsed as CSV: No columns to parse from file"). Four tests added; two earlier ones updated, since they asserted the single wording this deliberately splits. 27 on the module, suite 388 pass, the same 16 failures pre-existing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Brings the track3 work onto the personal branch now that it is synced with mimosa_v2. Three conflicts. llm_provider.py — track3's version of the temperature recovery, which turns out to agree with Martin: mimosa_v2 already falls back to 1.0 rather than stripping the parameter, and track3 does the same behind a named _SAFE_MAX_TEMPERATURE, plus a heuristic for gateways that return a bare 400 with no `param` field (the OpenRouter case). The "strip to None" variant on this branch predates both. Martin's _is_claude_model gate is preserved alongside track3's truncation- escalation state, since those are orthogonal. smolagent_factory.py — track3's capture of the last exception. Without it an agent that exhausts its retries leaves completed=False with no exception and the caller reports a timeout that never happened. perspicacite_client.py — a real hand-merge. This branch threads kb_name through _cache_key/_write_cache, which per-side KB selection requires: without it two knowledge bases share one cache entry. track3 instead carried a _SETTINGS global and _record() telemetry. Kept the kb_name threading, added the telemetry on both the success and failure paths, and took _SETTINGS["max_papers"] over the hardcoded 5. Suite: 395 pass; the same 16 failures are pre-existing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
New run-detail tab rendering the run from its structured record rather
than its own reporting. Backend: GET /api/runs/{id}/provenance serves the
transparency exporter's ASTRA capsule (decisions with alternatives and
universes; runs without one link to the family members that have one)
plus every asb_eval evaluation capsule naming the run (executor verdicts,
workspace provenance flags, pinned-instrument judge layer). Follows
store.py's defensive contract; capsule/eval roots overridable via
MIMOSA_CAPSULE_DIR / MIMOSA_EVAL_DIR. Frontend: ProvenancePanel with
decision rows (chosen green, alternatives dim), evaluation cards
(score-over-decided headline, per-criterion verdicts, flags), and the
judge block kept visually separate from the executor score.
…n feed
Four surfaces on top of the Provenance work:
Rendering (frontend/src/render/): shiki-highlighted code, collapsible
JSON tree, GFM markdown, CSV table previews, ANSI-stripped logs — wired
into the replay inspector, workspace previews, and the artifacts tab,
each with a raw toggle.
Evolution tab: the family replayed as an animated SVG lineage (reveal
per tick, play/step/speed/scrub, best-so-far badge chasing the
frontier, ancestry glow) with a per-node narration panel — score delta
vs parent, claims, QD/novelty, cost, selection log, and the textual
gradient. Served by GET /runs/{id}/evolution.
Atlas (/atlas, GET /atlas/{space}): every run PCA-projected with
lineage trails, colour modes, family filter, fleet time-replay,
zoom/pan, and a per-family point-by-point trajectory mode. Two spaces:
qd (Mimosa's behaviour descriptor — measured byte-identical within a
family, i.e. task-level, families coincide by construction) and
genotype (TF-IDF of the evolved workflow code, 113 runs, where
within-family drift is real). The trajectory bar says so and offers
the switch when a family's points coincide.
Live: the watcher now covers workflows, memory, runs_capsule and
evaluations roots and emits step_appended / llm_call_logged /
gradient_updated / evaluation_updated / astra_updated /
evaluation_capsule_updated. The run page auto-refreshes affected
panels and shows a live feed that diffs the ASTRA decision layer on
every capsule update.
Backend tests 57 -> 67; tsc, oxlint, vite build clean; new deps
shiki/react-markdown/remark-gfm/diff (frontend), numpy (backend).
lfnothias
changed the title
This was referenced Aug 26, 2026
Closed
Open
Collaborator
Author
|
Split into four independent slices against
The June commits (model tiers, verifier cost knobs, per-side KB grounding, sandbox imports) are not re-proposed here: model tiers were declined in #180, and the verifier knobs are still #181. This PR stays open as the umbrella for #198 and will be closed once the slices are in. |
Member
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Turns the Observatory into an examination instrument for self-evolving runs: provenance from the structured record, rich rendering of run content, an animated evolution replay, a descriptor-space atlas, and a live activity feed.
Provenance tab. A new
GET /api/runs/{id}/provenanceendpoint serves (a) the transparency exporter's ASTRA capsule for the run — decisions with their alternatives, rationale, and realised universes; a run without a capsule of its own links to the family members that have one — and (b) everyasb_evalevaluation capsule (eval_astra.yaml) that names the run: executor verdicts over the ASB card's own criteria, workspace provenance flags, and the pinned-instrument judge layer.ProvenancePanelrenders decision rows (chosen option highlighted, alternatives dimmed with exclusion reasons), evaluation cards with the score-over-decided headline, and the judge block kept visually separate from the executor score. Capsule and evaluation roots are overridable viaMIMOSA_CAPSULE_DIR/MIMOSA_EVAL_DIR.Rich rendering. A shared
render/module (syntax highlighting via shiki, collapsible JSON tree, GFM markdown, CSV preview, ANSI stripping) replaces raw<pre>dumps in the memory replay inspector, workspace previews, and run artifacts; every view keeps a raw toggle.Evolution replay.
GET /api/runs/{id}/evolutionjoins the lineage tree with per-node metrics, claim verdicts, costs, the selection log, and the textual gradient. The new Evolution tab animates the family generation by generation — play/step/scrub controls, ancestry highlighting, best-so-far marker, and a narration panel showing the score delta against the parent and the gradient that preceded each mutation.Descriptor atlas.
GET /api/atlas/{space}projects run descriptors by PCA, with lineage edges and per-family trajectory replay. Two spaces are exposed: the QD behaviour descriptor, and a TF-IDF genotype space over each run's evolved workflow code. The second exists because the QD descriptor proved byte-identical within a family (it embeds the task, not the evolved workflow), so within-family trajectories are degenerate in that space by construction; the UI states the limitation and offers the switch rather than masking it.Live view. A filesystem watcher over the workflow, memory, capsule, and evaluation roots streams semantic events over
WS /api/live(iteration completed, gradient written, ASTRA capsule updated, evaluation updated, …). Running runs get a live feed that diffs the ASTRA decision layer on each capsule write and auto-refreshes the workspace and provenance panels. Granularity is per-iteration, matching when the exporter writes; mid-iteration decision streaming would need a hook inastra_exporterand is left out of scope here.Verified against live runs: the capsule decisions and asb_eval verdicts of
20260822_183340_88cdd1ferender end to end; the evolution replay and both atlas spaces were exercised on the run-7 families (113 runs in genotype space); live events were confirmed by touching real artifacts and observing the corresponding feed lines. Backend tests 52 → 67;tsc,oxlint, andvite buildclean.