diff --git a/.cursor/rules/api-key-controls.mdc b/.cursor/rules/api-key-controls.mdc index 0862b80..ca9cca4 100644 --- a/.cursor/rules/api-key-controls.mdc +++ b/.cursor/rules/api-key-controls.mdc @@ -7,7 +7,7 @@ alwaysApply: true ## Overview -Enables cloud provider access with LM Studio fallback in default mode (OpenRouter API keys plus desktop OAuth providers such as OpenAI Codex and xAI Grok/SuperGrok) and OpenRouter-only operation in generic mode, plus boost controls and research metrics in the workflow panel. +Enables cloud provider access with LM Studio fallback in default mode (OpenRouter API keys plus desktop OAuth providers such as OpenAI Codex and xAI Grok/SuperGrok) and OpenRouter-only operation in generic mode, plus boost controls, research metrics, and the Assistant Memory Bank in the workflow panel. **Key Features:** - **Per-Role Cloud Provider Selection**: Each role independently uses LM Studio, OpenRouter, or a desktop-only OAuth/subscription provider where available (OpenAI Codex OAuth, xAI Grok OAuth, Sakana Fugu API key; default mode); generic mode remains OpenRouter-only. @@ -15,7 +15,8 @@ Enables cloud provider access with LM Studio fallback in default mode (OpenRoute - **Credential State Refresh**: Adding/removing OpenRouter keys or desktop OAuth/subscription credentials must immediately invalidate mounted settings panels so provider buttons and model lists refresh without tab navigation. Credential presence and model-list loading are distinct: a valid OpenRouter key stays enabled if `/models` transiently fails; failed OAuth/subscription model loads clear stale model lists and show provider-specific errors. - **SyntheticLib4 Access**: SyntheticLib4 backend/search scaffolding remains in place, but the user-facing connectivity pill is currently a `Coming soon` explainer rather than a ready/configuration surface. The modal explains the reciprocal proof-contribution/access model until production corpus access is enabled. - **Session History Memory**: The connectivity toggle defaults enabled for new users and maps to local MOTO/manual/LeanOJ proof-history memory used by Assistant workflow-memory search during brainstorming, writing, proof work, and LeanOJ solving. It is not raw provider transcript or chain-of-thought storage. Disabling it persists as non-secret runtime state, removes local proof-history corpora from Assistant retrieval, and must not delete proof records or alter internal retry/rejection/prompt memory. -- **Assistant Role**: Aggregator, Compiler, Autonomous Research, and LeanOJ expose one shared non-blocking Assistant LLM role per workflow surface for verified proof-memory support. Assistant selects up to 7 prior verified proof supports, never replaces validators or submitters, never blocks parent workflows, and is disabled when Session History Memory is disabled. Useful Assistant packs may be reused by two eligible receiver reads before the next refresh. True no-history targets are skipped because Assistant only performs proof-memory retrieval for now. Durable cooldown groups transient task IDs/roles by workflow run while preserving real source/session separation: repeated zero-useful retrieval backs off and may shut down for the run; repeated stagnant same-pack retrieval backs off without shutdown. Stale live packs/state clear only on explicit reset/clear or Session History Memory disable. User live activity should show normal Assistant retrieval result logs and explicit Assistant model-output failures, not skip/backoff/shutdown turns. +- **Assistant Role**: Aggregator, Compiler, Autonomous Research, and LeanOJ expose one shared non-blocking Assistant LLM role per workflow surface for verified Lean proof-history support. Autonomous papers-only runs suppress Assistant proof-memory scheduling and injection for the parent and its child Aggregator/Compiler roles without changing the user's persisted Session History setting. Otherwise supports are optional: Assistant cannot redirect or mathematically reinterpret an unrelated parent goal, and no useful proof support is a valid result. Assistant selects up to 7 prior verified proof supports, never replaces validators or submitters, never blocks parent workflows, and is disabled when Session History Memory is disabled. Useful Assistant packs may be reused by two eligible receiver reads before the next refresh. True no-history targets are skipped because Assistant only performs proof-memory retrieval for now. Durable cooldown groups transient task IDs/roles by workflow run while preserving real source/session separation: repeated zero-useful retrieval backs off and may shut down for the run; repeated stagnant same-pack retrieval backs off without shutdown. Top-level Stop clears live/latest pack state, while schema-validated SQLite ranking/goal/pack history can remain rebuildable; explicit Clear/reset or Session History Memory disable also clears durable run-scoped Assistant state. User live activity should show normal Assistant retrieval result logs and explicit Assistant model-output failures, not skip/backoff/shutdown turns. The WorkflowPanel Assistant Memory Bank may show the latest metadata-only up-to-7 pack as novelty-colored proof-history tiles; it must not expose Lean code through pack events. +- **Progressive Solution Path**: Before five cumulative accepted brainstorm events in a top-level run it is absent from prompts and UI. After activation, only existing semantic validators may optionally propose material updates through separately parsed normal-response JSON; this never changes or blocks the primary decision. Updates apply only after transactional serial review by a dedicated role configured exactly from Main Submitter 1. Hard provider/config/context failures enter visible user-repair state rather than retrying forever; after settings repair, the user explicitly retries the generation-fenced proposal through `POST /api/workflow/solution-path/resume`. Stop/crash preserves state, while Clear/new-run is the only count reset and is serialized against reacquisition. - **Startup provider requirement**: OAuth providers are supplementary role providers, not a standalone startup path, because RAG embeddings route through LM Studio, OpenRouter, or generic-mode FastEmbed. First-run startup and workflow start preflights must require OpenRouter, generic FastEmbed, or LM Studio with an embedding model available before OAuth-only role selection is allowed. - **OpenRouter Auto-Fill**: OpenRouter selectors fetch provider endpoint metadata and compute host-aware context/output settings from a capable endpoint set. Auto mode ignores known weak hosts (currently Venice) and low/missing-cap outliers before computing context/max-output; manual host selection uses that exact host and its largest exposed endpoint output cap. - **OAuth Auto-Fill**: OAuth model selectors auto-fill only from provider model metadata, documented provider-specific limits, or curated provider-backed public aliases. Do not invent generic fallback context windows for unknown OAuth models; preserve current settings when metadata is unknown. GPT-5.5 Codex uses the Codex 400K product window, not the 1M regular API window; Codex Spark high is exposed as a curated alias for Codex Spark with high reasoning and its documented 128K window. Grok/SuperGrok OAuth uses xAI model metadata when available and may expose known Grok subscription limits only for known model IDs; model listings must filter xAI catalog entries that are not accepted by the OAuth chat-completions route, such as multi-agent-only models. @@ -92,7 +93,7 @@ Enables cloud provider access with LM Studio fallback in default mode (OpenRoute - OpenAI-compatible assistant text extraction is centralized in `response_extraction.py`: prefer `message.content`, then compatibility fallback fields such as `reasoning`/`thinking` when content is empty so local reasoning-only endpoints still work under one auditable policy. - Raw provider/model transport output must never be replayed into MOTO retry prompts, feedback memory, accepted memory, RAG, or durable context. Conversational retries are required, but failed-output context must first pass `sanitize_model_output_for_retry_context()` so only reusable visible answer text remains. The sanitizer strips known private thought/channel/control tokens only as transport scaffolding outside visible JSON/string content, not ordinary visible Lean/math/operator syntax such as `<|` or literal visible marker text such as `<|channel>final` / `` inside content. - Parser exception strings that are inserted into retry prompts must not contain raw response excerpts; raw excerpts are allowed only in logs/observability surfaces that are never reused as model context. -- Observability surfaces must default to metadata/previews with secret redaction. Provider keys, URL query keys, Wolfram query/result text, and full prompt/response bodies must not be persisted or broadcast unless an explicit trusted debug path opts in. Legacy full-payload log fields are scrubbed from persisted API logs on logger startup. +- Persisted observability logs are always bounded metadata/secret-redacted previews and never contain full prompt/response bodies. Explicit desktop debug inspection may retain bounded full payloads only in process memory; generic mode cannot enable it. Legacy full-payload log fields are scrubbed on logger startup. - Tool-call assistant/tool protocol turns are the only exception where exact assistant content/structure may need preservation; ordinary JSON retry assistant turns are not tool protocol turns and must use sanitized retry context. - Generic mode must normalize or reject LM Studio role configs and must never fall through to `lm_studio_client.generate_completion()`, even if a direct API caller submits legacy `provider="lm_studio"` or an LM fallback value. - Desktop OAuth providers and Sakana Fugu (`sakana_fugu`) are distinct from regular provider API-key billing. OpenAI Codex OAuth (`openai_codex_oauth`) uses ChatGPT/Codex account tokens against `https://chatgpt.com/backend-api/codex`; xAI Grok OAuth (`xai_grok_oauth`) uses current `auth.x.ai` PKCE login with `grok-cli:access` / `api:access` scopes and subscription tokens against the xAI OpenAI-compatible chat-completions API; Sakana Fugu (`sakana_fugu`) uses the desktop keyring-stored subscription API key against Sakana's OpenAI-compatible `/responses` and `/chat/completions` endpoints. xAI Console API keys are a separate pay-as-you-go/credit path, not the subscription-backed OAuth path. OAuth tokens are stored through chunked OS-keyring entries, Sakana uses a single keyring entry, provider status is non-secret, callback listeners are loopback-only and released after pending login completion/expiry, and these desktop-only providers remain unavailable in generic mode. Hosted/generic settings must not poll desktop OAuth/Sakana status/model endpoints when `/api/features` marks those capabilities unavailable. @@ -106,7 +107,7 @@ Enables cloud provider access with LM Studio fallback in default mode (OpenRoute - This includes: aggregator submitters/validator/Assistant, compiler Writing/Rigor & Proofs/validator/Assistant, autonomous agents/Assistant, Tier 3 final answer agents, and LeanOJ roles/topic/brainstorm submitters/Assistant - Role configs must preserve `supercharge_enabled` when copied into proof snapshots, manual proof helpers, child Aggregator/Compiler coordinators, and LeanOJ grouped roles. - **Proof agents (Part 3/manual, optional)** use the visible Rigor & Proofs Submitter settings for proof-solving submitter work. Internal proof role IDs are still configured through `api_client_manager.configure_role()` by copying from the `ProofRuntimeConfigSnapshot` (Rigor & Proofs submitter, validator, optional Assistant) captured by `autonomous_coordinator._build_proof_runtime_config_snapshot()` and persisted via `research_metadata.set_proof_runtime_config()`, or supplied directly on manual `POST /api/proofs/check`. Manual checks require `lean4_enabled=True` and either a stored or request-provided runtime snapshot. -- Assistant is the preferred owner of routine proof-history memory retrieval for eligible non-validator, non-critique flows. Assistant proof-memory injection is centralized in `APIClientManager`, optional, budget-bounded, excluded from validators/Assistant-like roles, and consumed only by eligible receiver calls. It watches the current prompt/phase/target, searches proactively against verified proof corpora, ranks a diverse up-to-7 memory-support pack with cached target/goal reuse and visit-count exploration, reuses useful packs for two eligible receiver reads before refresh, and never blocks the parent workflow. Run-scoped cooldown prevents repeated empty/stagnant retrieval from calling Assistant every turn; ordinary stop preserves that state, while explicit clear/reset removes it. In-role `search_lean_proofs` calls should be explicit legacy/debug or narrow emergency-repair paths only, not the normal retrieval path. +- Assistant is the preferred owner of routine proof-history memory retrieval for eligible non-validator, non-critique flows. Each refresh deterministically fuses concurrent enabled-local-history, versioned canonical theorem/code identity → composite occurrence/run neighborhood, and SyntheticLib4 lanes without quotas; publisher integrity hashes remain separate. Exact neighborhoods are bounded, non-recursive, and current-run/session excluded. A typed standalone exact-duplicate-emphasis purpose is persisted on the exact cross-run overlay occurrence and excluded before fusion; ordinary `duplicate_novel` occurrences stay public-search/Assistant eligible. Reused packs are policy-versioned and run/session re-filtered; mixed supports drop marked occurrences individually and survive when an eligible occurrence remains. Reciprocal-rank evidence is aggregated per artifact before an at-most-64 → 21 reduction; the normal configured-provider path makes at most one non-Supercharged Assistant selection call for up to 7 supports, while unavailable/no-history/no-candidate/cooldown paths may make none. Two concrete receiver reads, run-scoped cooldown/provider behavior, and narrow emergency-only in-role search remain unchanged. **Boost Mode Priority** (`should_use_boost(task_id)`): 1. Always Prefer Boost: `boost_always_prefer=True` → True @@ -137,7 +138,8 @@ Enables cloud provider access with LM Studio fallback in default mode (OpenRoute #### SyntheticLib4 Access And Proof Search (planned) - SyntheticLib4 integration is an authorized proof-corpus/search surface, not a model route. MOTO may use hosted APIs, downloaded snapshots, and deltas, but must not clone the private Lean repository or scrape website pages. - Desktop/default credentials use `secret_store.py`; hosted/generic credentials remain env-injected or in process memory. Snapshot files, runtime settings, prompts, and default logs must never contain SyntheticLib4 tokens or API keys. -- The initial unified proof-search backend indexes active and archived canonical MOTO proof records plus SyntheticLib4 mock/offline records through batched local SQLite/FTS rebuilds; search responses enforce the 7-proof combined result cap before workflow/tool integration. Assistant may gather a wider local candidate pool, but final model-visible proof-support packs remain capped at 7. SyntheticLib4 mock/offline mode must keep working when test fixtures are not packaged by using bundled/built-in fallback records until a real data-root snapshot is available. +- The initial unified proof-search backend indexes active and archived canonical MOTO proof records plus an authorized active SyntheticLib4 snapshot through batched local SQLite/FTS rebuilds; search responses enforce the 7-proof combined result cap before workflow/tool integration. Assistant may gather a wider local candidate pool, but final model-visible proof-support packs remain capped at 7. Normal runtime reports zero SyntheticLib4 records until a snapshot is explicitly activated; fixture records are test-only and never a production fallback. +- Duplicate-novel proof records are redundant local occurrences that remain SyntheticLib-tradable/searchable; unified proof search and Assistant retrieval must include them and `not_novel` records unless explicitly filtered out. - Proof-search service calls must centrally filter disabled corpora for REST, tool, and Assistant paths; disabling Session History Memory removes local MOTO/manual/LeanOJ corpora, and disabling SyntheticLib4 removes that corpus even if a client requests it explicitly. - The shared `search_lean_proofs` tool adapter supports overview/search/hydrate/local usage-attestation actions over the same service. Proof formalization exposes the tool through a bounded provider tool-call loop where supported, while preserving the existing prefetch path for providers that do not emit tool calls; local per-attempt exclusions avoid exact repeat results. Proof formalization records local `model_visible_context` attestations when full SyntheticLib4 Lean code is injected into a successful formalization prompt; whole-code-use submission to SyntheticLib4 remains a later live-service step. - Local snapshot search remains available when lawfully downloaded and allowed by the recorded license/terms, but refresh, hosted retrieval, user-proof browsing, sharing, redistribution, and public serving require active authorization unless enterprise rights explicitly allow more. @@ -264,9 +266,10 @@ Workflow, boost, fallback/reset, provider pause/resume, hung-connection, rate-li ## Free Model Cooldown & Rotation System -**Singleton:** `free_model_manager` in `backend/shared/free_model_manager.py`. Two global settings (both default ON): +**Singleton:** `free_model_manager` in `backend/shared/free_model_manager.py`. Two global settings (both default OFF): - `looping_enabled` — rotate to next available free model on rate limit (highest context first) - `auto_selector_enabled` — fall back to `openrouter/free` when all free models are exhausted; context/max-output still come from configured model metadata or user settings, not a hidden 131K default +- Free-model rotation and `free_only` model lists are restricted to text-in/text-out chat-capable models; non-text image/audio free models must not be offered as role/fallback targets. **Rotation chain** (in `api_client_manager._try_free_model_rotation()` called from RateLimitError handler; keep optional `tools` / `tool_choice` passed through when that helper is used): 1. If `looping_enabled`: **iterate through ALL** non-rate-limited free models (highest context first) using `tried_models` set to avoid re-trying. On each `RateLimitError`, refresh rate-limited dict and continue to next model. On `CreditExhaustionError`, stop looping. @@ -294,7 +297,7 @@ Workflow, boost, fallback/reset, provider pause/resume, hung-connection, rate-li **WebSocket Events:** Free-model rotation, auto-selector use, free-model exhaustion/retry, and account-exhaustion states should be visible to the UI; exact internal event names are not rule-level invariants unless consumed by the hosted wrapper or frontend contract. -**Frontend:** Aggregator, Compiler, and Autonomous settings expose two OpenRouter free-model fallback controls in their settings panels (currently grouped in the OpenRouter fallback area rather than necessarily adjacent to "Free models only"). Both default checked, hydrate from the backend singleton, and persist as non-secret runtime settings under the active data root plus localStorage UI state. LeanOJ settings currently expose the "Free models only" filter but not the looping/auto-selector controls. +**Frontend:** Aggregator, Compiler, and Autonomous settings expose two OpenRouter free-model fallback controls in their settings panels (currently grouped in the OpenRouter fallback area rather than necessarily adjacent to "Free models only"). Both default unchecked, hydrate from the backend singleton, and persist as non-secret runtime settings under the active data root plus localStorage UI state. LeanOJ settings currently expose the text-chat-only "Free models only" filter but not the looping/auto-selector controls. --- @@ -302,6 +305,8 @@ Workflow, boost, fallback/reset, provider pause/resume, hung-connection, rate-li **Secure backend storage (OS keyring — default mode):** OpenRouter global API key and Wolfram Alpha API key persist via `backend/shared/secret_store.py` using the OS keychain/keyring. The keyring service name is derived from `MOTO_SECRET_NAMESPACE`; `None` means the shared desktop service `MOTO-Autonomous-ASI`. `moto_launcher.py` MUST keep plain/default launches on the shared default identity (`backend/data`, `backend/logs`, no keyring namespace, no frontend storage prefix, frontend port 5173) so backend keys, browser profiles/prompts, sessions, and proofs remain visible. Never let port availability, Windows `TIME_WAIT`, stale `.moto_last_instance.json`, or Lean/LM startup timing create a new namespace or data root for a plain relaunch; concurrent default launches must be blocked by active-instance state and a data-root runtime lock. +**Data-root ownership:** FastAPI lifespan holds an OS-backed lease for the active data root before mutable-store initialization and releases it only after shutdown. Launcher `.moto_runtime.lock` remains advisory process metadata and is not the authoritative backend lease. + **Startup key detection:** `backend/api/main.py` restores desktop credentials before serving `/api/openrouter/api-key-status`. Expensive optional startup work (Lean 4 warm start, Mathlib cache, etc.) must not block the FastAPI lifespan; run it in the background and clean up subprocesses on cancellation. Frontend startup state must treat unreachable key-status as `unknown`, not `has_key=false`, so it never opens the setup modal or shows a missing-key state over a persisted key. **Hosted generic mode (no keyring):** Provider keys are env-injected at sandbox launch and/or set via proxied MOTO API routes. `secret_store` persistence is bypassed; keys live in sandbox memory only. Re-injection required after sandbox recreation. `OPENROUTER_API_KEY` env var auto-loaded during lifespan if present. @@ -310,4 +315,4 @@ Workflow, boost, fallback/reset, provider pause/resume, hung-connection, rate-li **localStorage:** Key families include workflow/UI preferences (`workflow_panel_collapsed`, `developerModeSettingsEnabled`, `banner_shimmer_enabled`, `startup_provider_choice`, high-score critique seen keys), settings/profile keys (`aggregatorConfig` / `aggregator_settings`, `compiler_settings`, `autonomous_research_settings`, `autonomous_research_profiles`, `leanoj_solver_settings`, `leanoj_solver_profiles`, boost modal settings), bounded live-activity histories, prompt helpers, and related free-model/Supercharge fields. Persisted frontend settings must avoid secret-like fields and bulky refetchable provider metadata; quota recovery may clear refetchable caches while keeping essential settings. Active app mode and tab state are not persisted; a fresh frontend mount starts on the autonomous main interface. Browser storage namespacing is driven by `VITE_MOTO_STORAGE_PREFIX`; launch/control-plane config may supply `MOTO_FRONTEND_STORAGE_PREFIX` and project it into the frontend env. -**Session (in-memory):** fallback state per role, boosted task IDs, boost next count, boosted categories, completed task IDs, free model manager state, and any explicit Boost override key. Boost override keys must never be persisted to `boost_state.json`; legacy plaintext keys are ignored/scrubbed on load. Boost logs and non-secret boost routing state persist under the active instance data root (`boost_api_log.txt`, `boost_state.json`) and are merged into the main API call log view. API call logs store previews/metadata by default; full prompt/response payload persistence is debug opt-in only, and provider/model error logs must report shape/status metadata instead of raw response bodies. +**Session (in-memory):** fallback state per role, boosted task IDs, boost next count, boosted categories, completed task IDs, free model manager state, any explicit Boost override key, and optional bounded desktop debug payloads. Boost override keys must never be persisted to `boost_state.json`; legacy plaintext keys are ignored/scrubbed on load. Boost logs and non-secret boost routing state persist under the active instance data root (`boost_api_log.txt`, `boost_state.json`) and are merged into the main API call log view. API call logs persist previews/metadata only, and provider/model error logs must report shape/status metadata instead of raw response bodies. diff --git a/.cursor/rules/hosted-web-contract.mdc b/.cursor/rules/hosted-web-contract.mdc index e7834b5..74f9a98 100644 --- a/.cursor/rules/hosted-web-contract.mdc +++ b/.cursor/rules/hosted-web-contract.mdc @@ -79,6 +79,8 @@ Implementation: centralized in `middleware.py` (HTTP) and `websocket.py` (before ## `/api/features` Endpoint (Both Modes) +Build 5 v73 completes the typed Progressive Solution Path snapshot/retry and compact lifecycle events, adds stale-safe repair UI behavior, and persists a narrow standalone exact-duplicate-emphasis proof purpose with versioned Assistant-pack filtering that preserves eligible mixed occurrences. v72 makes path activation parent-owned and monotonic, routes review through exact Main Submitter 1 settings, applies edits transactionally, and adds durable repair state. + Build 0 lands the public identity subset first. Returns: ```python { @@ -95,9 +97,11 @@ Build 0 lands the public identity subset first. Returns: } ``` -The current Build 5 runtime preserves the four identity fields while exposing the stable capability flags above. `proof_downshifted` is a proof workflow event for Lean-accepted proofs preserved under a narrower actual theorem statement, not a `/api/features` field. Later hosted work may extend `/api/features` with additional capability flags such as `max_submitters` and `tier3_available`, but the existing fields above remain stable and `api_contract_version` must bump when that happens. +The current Build 5 runtime preserves the four identity fields while exposing the stable capability flags above. Build 5 v67 adds safe configured/effective model-route identity to provider-side context-overflow activity, including terminal fallback and Boost/free-rotation routes without exposing secrets. v66 distinguishes nonfatal workflow-scoped `proof_context_overflow` activity from fatal workflow `context_overflow_error`, scopes overflow activity by workflow, preserves attributed activity across reload, and prevents duplicate fatal stop entries. Build 5 v65 adds the fast typed `GET /api/workflow/solution-path` snapshot and compact solution-path update events for the user-facing route viewer. `proof_downshifted` is a proof workflow event for Lean-accepted proofs preserved under a narrower actual theorem statement, not a `/api/features` field. Later hosted work may extend `/api/features` with additional capability flags such as `max_submitters` and `tier3_available`, but the existing fields above remain stable and `api_contract_version` must bump when that happens. + +Build 5 v64 adds canonical `search_id`/`run_id` proof hydration query parameters and race-safe frontend proof-detail selection. v63 adds bounded paginated metadata-only Assistant support lineage retrieval, one-vote-per-artifact-per-lane fusion, archived-manual Lean hydration, and run-aware proof-library navigation/control persistence. v62 scoped exact-proof neighborhoods to composite run provenance, aggregated deterministic cross-lane rank evidence without quotas, and extended bounded Assistant lineage metadata while keeping complete backend provenance. It also unified response-time legacy proof provenance and federated proof-search query controls. v61 added typed proof-library/detail/certificate and latest-Assistant-pack schemas, strict proof category validation, and archive-keyed autonomous/manual JSON and Lean certificate exports with stable run/prompt/novelty provenance. v60 made private exact-proof duplication search metadata only, preserved independently novelty-ranked occurrences with canonical prompt/stable run provenance, and added redacted multi-lane Assistant counts with bounded staged reduction (at most 64 → up to 21 → up to 7) and one selection call. -Current Build 5 contract notes: v54 includes `assistant_proof_pack_failed` for Assistant proof-memory model-output/selection failures, which should display as live activity errors instead of successful empty proof-pack updates. Autonomous proof provider pauses with `reason="openrouter_credit_exhaustion"` remain credit-reset prompts; `reason="transient_provider_error"` is an automatic retry pause, not a credit notification. v53 includes desktop-only Sakana Fugu direct subscription API access (`sakana_fugu_available`, `/api/cloud-access/sakana-fugu/*`, Responses-first generation with chat-completions fallback, and `sakana_fugu_error` notifications), Codex OAuth `usage_limit_reached` cooldown metadata/notifications, and Assistant proof-pack OAuth-cooldown selection modes (`cached_oauth_cooldown`, `deterministic_oauth_cooldown`). v52 includes durable internal Assistant proof-memory cooldown/shutdown/no-history WebSocket events (`assistant_proof_memory_unavailable`, `assistant_proof_memory_cooldown`, `assistant_proof_memory_shutdown`), which should stay hidden from user live activity. +Current Build 5 contract notes: v59 adds attributed context-overflow activity. v57 adds proof-library category filtering (`category=novel|duplicate_novel|not_novel|all`); under v71, `duplicate_novel` means a searchable/tradable cross-run exact-identity overlay applied only after an independent novel judgment, with independent fields preserved and highest-priority injection excluded. v56 adds `.lean` uploads and logical cleanup; v55 adds metadata-only latest Assistant packs; v54 distinguishes Assistant selection failures. Earlier provider/OAuth pause and notification behavior remains unchanged. Earlier Build 5 contract additions include connectivity toggles/status, SyntheticLib4 and local proof-search routes, scoped proof/manual-history routes, provider/OAuth notification recovery, manual prompt recovery, proof-output allowed-output fields, and direct-context overflow events. Keep exact legacy details in code/tests/git history rather than expanding this always-injected rule. @@ -150,23 +154,25 @@ Sandbox is API-only. The MOTO React frontend is NOT served from the hosted sandb - Desktop default: `secret_store.py` uses OS keyring, restored on startup - Hosted generic mode: provider keys are env-injected at sandbox launch and/or set via proxied MOTO API routes. `secret_store` persistence is bypassed; keys live in sandbox memory only. Re-injection required after sandbox recreation. - Generic-mode OpenRouter and Wolfram routes update runtime memory only; they do not write to or clear the desktop keyring. +- Generic mode always persists API-call observability as bounded secret-redacted previews/metadata only; the desktop volatile full-payload debug cache remains disabled even if its environment flag is set. - Control plane NEVER stores, logs, or persists user provider keys in its own database ## Data Persistence - `backend/data/` is the default desktop working set +- ChromaDB is derived cache state, not authoritative research state. Windows desktop replaces prior-process Chroma state before the first native open and lazily rebuilds sources; hosted Linux retains persistent-cache behavior. One backend process holds an OS-backed lease for each active data root for its full lifespan. - Hosted: `MOTO_DATA_ROOT=/app/backend/data` so Blaxel storage mounts to one unambiguous path - Non-secret `runtime_settings.json` also lives under the active data root; it may persist runtime knobs, never provider keys or prompt/response payloads - ChromaDB SQLite files stay on Blaxel sandbox storage (local file semantics required) - SyntheticLib4 snapshots and unified proof-search indexes are data-root artifacts when implemented. They must be activated only after manifest/hash/index checks pass, and a failed refresh must leave the previous active snapshot usable. - Sandbox recall/resume returns the same filesystem state; redeploy/recreate advances to the newest image -- Uploads: server-side enforcement of `.txt` only, 5 MB max, filename sanitization, path traversal rejection +- Uploads: server-side enforcement of `.txt` and `.lean` UTF-8 text files only, 5 MB max, filename sanitization, path traversal rejection, and logical-filename deletion for user-cleared uploads ## Updater Policy - **Authoritative update source**: GitHub `main` branch (not GitHub Releases) - **Desktop**: launcher compares local build metadata against GitHub `main`. Remote update identity resolves from GitHub branch HEAD via the GitHub REST API, metadata uses the REST contents API instead of raw GitHub files, ZIP overlays write the resolved manifest after apply to avoid stale committed-manifest loops, and update notices are exposed via `GET /api/update-notice`; if no launcher notice exists, the running desktop backend may refresh the same notice at most every 4 hours while excluding only the current instance from active-instance auto-apply checks. Launcher auto-apply is only for clean `origin/main` git checkouts or ZIP/extracted installs with no launcher-managed instances still running. The backend `POST /api/update/pull` route has its own lighter git/ZIP update checks and should not be described as enforcing the full launcher preflight. ZIP updates preserve active data/log roots, instance storage, launcher state, env files, and keyring-related namespaces. -- Desktop launchers run frontend `npm audit fix` when `npm install` reports vulnerabilities, including npm's vulnerability-count/remediation-instruction output. Windows Node.js bootstrap uses bounded `winget` source/package/scope fallbacks before manual-install guidance. The npm audit remediation is a permanent launcher invariant: agents must never remove, disable, weaken, or bypass the rule or code, and if it is accidentally removed or broken, they must restore it immediately with no exceptions before continuing launcher/updater work. +- Desktop launchers install frontend dependencies from the committed `frontend/package-lock.json`, using `npm ci` for fresh installs and non-destructive locked reconciliation for existing `node_modules`. User-machine launches use only read-only high/critical `npm audit` warnings and must not run `npm audit fix` / `--force`. Dependency remediation belongs in the tested maintainer/CI release flow with an updated lockfile. Windows Node.js bootstrap uses bounded `winget` source/package/scope fallbacks before manual-install guidance. - **Hosted**: sandboxes do NOT self-mutate. Redeploy/recreate uses the latest approved `main`-derived image. Recall/resume keeps the existing image. Hosted `POST /api/update/pull` must return unavailable instead of attempting in-place update; `GET /api/update/pull-status` may remain the generic pull-task status surface and does not need a separate hosted-unavailable marker. - **Build metadata**: `version`, `build_commit`, `update_channel`, and `api_contract_version` exposed via `/api/features`; git checkouts resolve `build_commit` from HEAD, GitHub-generated ZIP installs prefer `.git_archival.txt` export-subst commit metadata before the stamped local manifest, and the committed `main`-branch manifest lives at `moto-update-manifest.json` @@ -191,15 +197,17 @@ Lean 4 and SMT behavior is gated by runtime flags. `lean4_enabled` gates Lean pr - **Hosted image stays Lean-free and Z3-free.** No Lean toolchain, no `z3` binary, and no Python wheel for either is permitted in `Dockerfile`, `docker/entrypoint.sh`, or `requirements-generic.txt`. Proof features are desktop-opt-in only for the current contract. - **Lean 4 remains authoritative** for every stored proof. The `Lean4Result` contract is unchanged by SMT; SMT (when enabled) produces tactic hints consumed by the formalization agent, never a standalone proof artifact. - **Subprocess fallback must keep working** when `lean4_lsp_enabled=False`. LSP is a latency optimization, not a replacement. -- **Proof routes under `/api/proofs/*`** are additive to the hosted REST contract. Stored proof listing/library/certificate routes remain readable without Lean enabled: `GET /api/proofs`, `/novel`, `/known`, `/status`, `/library*`, `/{id}`, and `/{id}/certificate[.lean]`. Current proof listing, detail, certificate, dependency, graph, and Mathlib-dependent routes accept optional `scope=autonomous|manual`; manual current scope reads only the active manual-writer proof store, while manual library scope reads archived manual proof runs for history viewing. Lean-derived operations (`POST /api/proofs/check`, `GET /api/proofs/{id}/dependencies`, `/graph`, `/mathlib/{lemma_name}/dependents`) require `lean4_enabled`; `POST /api/proofs/settings` and `POST /api/proofs/cleanup-known-from-files` are unavailable in hosted generic mode. -- **LeanOJ routes** are additive to the hosted REST contract: start (which resumes matching saved progress when available), stop, status, clear, skip-brainstorm, force-brainstorm, master-proof draft/edit summaries, current-run proofs, and cross-session proof library endpoints live under `/api/leanoj/*`. +- **Proof routes under `/api/proofs/*`** are additive to the hosted REST contract. Stored proof list/detail/library/certificate records expose stable run/prompt provenance, display novelty, independent novelty tier/reasoning, exact-duplicate proof/run provenance, and versioned canonical theorem/Lean hashes. Legacy records receive safe response-time defaults without mutation. Read routes remain available without Lean; current versus archived manual scope and Lean-gated dependency/check routes retain their existing behavior. +- **LeanOJ routes** are additive to the hosted REST contract: start (which resumes matching saved progress when available), stop, status, clear, skip-brainstorm, force-brainstorm, master-proof draft/edit summaries, current-run proofs, and historical completed-run proof-library endpoints live under `/api/leanoj/*`. `GET /api/leanoj/library` excludes the currently loaded run; that run is exposed only by `GET /api/leanoj/proofs`. - **Creativity Emphasis Boost** is an optional developer-gated start-request field (`creativity_emphasis_boost_enabled`) for Aggregator, Autonomous Research, and LeanOJ; accepted/rejected brainstorm WebSocket payloads may include `creativity_emphasized`, and prompt-budget overflow falls back to the normal prompt for that slot. - **Pruned Stage 2 paper routes** are additive: pruned papers are removed from model context/RAG but remain downloadable under `/api/auto-research/paper-history/pruned*`; hard deletion is limited to explicit delete-all-pruned endpoints. -- **WebSocket progress events** for LeanOJ, compiler critique, provider/OAuth failures, and proof workflows are part of the web-surface contract only when consumed by the hosted wrapper or frontend. Keep them descriptive and stable enough for UI state, but avoid treating every internal progress notification as a permanent rule-level invariant. OAuth provider failures tell desktop users to reconnect the provider in OpenRouter/OAuth after unrecoverable auth/model-call failure. Autonomous proof checkpoint progress events may include `proof_round_index` and `proof_max_rounds`; `proof_verified` must only emit after proof registration/reuse and include `proof_id`; proof novelty and duplicate-registration events include `novelty_tier` and `novelty_reasoning` for live activity display. Manual proof events must remain isolated from autonomous proof activity/notifications/graphs unless event payloads become explicitly scoped. +- **WebSocket progress events** for LeanOJ, compiler critique, provider/OAuth failures, and proof workflows are part of the web-surface contract only when consumed by the hosted wrapper or frontend. Keep them descriptive and stable enough for UI state, but avoid treating every internal progress notification as a permanent rule-level invariant. OAuth provider failures tell desktop users to reconnect the provider in OpenRouter/OAuth after unrecoverable auth/model-call failure; provider max-output truncation before usable Lean code should appear as a failed proof attempt, not as an unrecoverable provider failure. Autonomous proof checkpoint progress events may include `proof_round_index` and `proof_max_rounds`; `proof_verified` must only emit after proof registration/reuse and include `proof_id`; proof novelty and duplicate-registration events include `novelty_tier` and `novelty_reasoning` for live activity display. Manual proof events must remain isolated from autonomous proof activity/notifications/graphs unless event payloads become explicitly scoped. - **Proof certificate exports stay text-based** (`.lean` source + JSON metadata). No binary-only proof artifacts. - **Proof runtime config snapshot** (`ProofRuntimeConfigSnapshot`) is persisted via `research_metadata` and may also be supplied directly on manual `POST /api/proofs/check`; required state is `lean4_enabled=True` AND either a stored or request-provided snapshot. - **SyntheticLib4 / MOTO proof search** is additive proof-history/corpus navigation. The shared `search_lean_proofs` tool adapter returns bounded retrieved proof context and provenance; it must not bypass MOTO's proof registration, Lean/integrity gates for MOTO-generated artifacts, or existing proof-runtime disablement in hosted generic mode. +- **Proof-search navigation routes** include `POST /api/proof-search/reindex`, metadata-only latest Assistant pack retrieval, and bounded paginated Assistant support lineage. Detail hydration remains corpus-toggle-aware and may use canonical `search_id` plus optional `session_id`/`run_id`. - **Assistant memory-support role** is an additive workflow/settings surface. It is one shared non-blocking LLM role per workflow surface, not a per-lane submitter clone; validators never receive Assistant context and parent workflows never wait for it. Assistant may provide up to 7 verified proof supports, reuse useful packs for two eligible receiver reads before refresh, and skip true no-external-history targets with `assistant_proof_memory_unavailable` because it only performs proof-memory retrieval for now. Durable cooldown is keyed to stable run scope, grouping transient task IDs/roles while keeping real source/session IDs separate; it emits internal `assistant_proof_memory_cooldown` during zero-useful or stagnant backoff and internal `assistant_proof_memory_shutdown` only when repeated zero-useful retrieval disables retrieval for the run scope. Stagnant same-pack retrieval must not shut down. User live activity should not show skip/backoff/shutdown turns; show normal Assistant retrieval summaries and explicit Assistant model-output failures. Session History Memory disabled disables Assistant and prevents stale pack injection. REST/WebSocket/profile schema additions for Assistant require this contract, `/openapi.json`, and `api_contract_version` to update in the same merge. +- **Progressive Solution Path** is one run-scoped advisory artifact for Manual Aggregator→Compiler, Autonomous, or LeanOJ. It remains absent before five monotonic parent-owned accepted brainstorm events. Only semantic validators originate optional material proposals; Main Submitter 1 review is exact-role-routed, serial, transactional, and non-blocking. `GET /api/workflow/solution-path` exposes fast typed ownership/lifecycle/queue/repair state; authenticated `POST /api/workflow/solution-path/resume` explicitly retries one repair-required proposal using run ID, proposal ID, and lifecycle generation. Compact path events carry run/generation identity and no raw model output. Stop/crash preserves state; generation-serialized Clear/new-run is the only reset. - **`api_contract_version` bumps** apply the same way to proof additions as to the base contract: any new proof route or event added after Build 5 must bump the contract version in the same merge. ## Hosting Ownership diff --git a/.cursor/rules/json-prompt-design.mdc b/.cursor/rules/json-prompt-design.mdc index c19d090..6f3bab4 100644 --- a/.cursor/rules/json-prompt-design.mdc +++ b/.cursor/rules/json-prompt-design.mdc @@ -1,2825 +1,44 @@ --- -description: JSON prompt schemas and formatting guidance for MOTO role interactions -alwaysApply: false +description: Protects MOTO prompt wording and defines cross-cutting JSON response invariants +alwaysApply: true --- -# Enhance AI Role Prompts with Complete Context Assembly +# Prompt Engineering and JSON Contract Protection -**VARIANT: MOTO - Math Variant V1** -**Focus:** Mathematical problem-solving, proofs, and theoretical analysis (not engineering applications) +## Explicit Consent Required -## Overview +MOTO's production prompt language is carefully engineered workflow behavior. Do not edit, rewrite, shorten, generalize, reformat, reorganize, or “improve” production prompt wording unless the user explicitly consents to and directs that prompt-language change in the current task. -This plan shows the complete prompt structure sent to each LLM for the **Math Variant** - focused on mathematical expositions, proofs, and theoretical analysis. +Permission to change adjacent code, schemas, models, validators, workflows, documentation, or rules is not permission to alter prompt wording. If an explicitly requested behavior change necessarily requires a prompt change, explain that dependency and obtain explicit user direction before editing the prompt. Once authorized, make only the requested minimum change and preserve all unrelated wording. -## Prompt Engineering Principles +Production prompt builders together with their parsers/models and executable tests are authoritative for exact wording and JSON contracts. Rules contain only durable high-level invariants; never copy complete production prompts, per-role schema catalogs, full response examples, tool schemas, or event payloads into rule files. -These principles prevent rejection loops and ensure models learn from feedback: +## Research Direction and Evidence -### 1. CONCRETE FORMAT EXAMPLES -Prompts that are prone to rejection loops or have complex structural requirements should include: -- ✅ **CORRECT format examples** with visual indicators (✓ checkmarks, green indicators) -- ❌ **WRONG format examples** with explanations of why they're invalid -- 🔧 **FIX instructions** showing how to correct common errors +- Ordinary research roles aggressively seek the strongest credible, genuinely novel solution to the user's exact objective. They prefer whole-question attacks, using the contribution form and claim-type rigor appropriate to the domain; necessary partial work must visibly advance the complete objective. +- AI-generated databases, accepted submissions, papers, outlines, retrieved history, and prior model responses are working evidence, not authority. Prompts must require appropriate independent verification, provenance, uncertainty, and skepticism rather than trusting content because MOTO previously accepted or produced it. -**Example Structure**: -``` -🔍 CORRECT FORMAT: +## JSON Contract Design -[Concrete example with proper formatting] +- Give each role and mode one explicit response contract containing only fields that its consumer accepts. Do not casually combine mutually exclusive mode schemas into a superset. +- Discriminator fields and dependent payloads must agree. A positive action requires its defined non-empty payload and valid operation fields; a decline/no-op uses that mode's defined empty or null form. A completion signal cannot substitute for required content. +- Batched outputs return exactly one ordered, stably identified result for every input. Preserve independent item assessment before any batch-level redundancy resolution. +- Parse optional cross-cutting extensions separately from the primary result. Omission is normal, and a malformed optional extension must not reject, alter, or invalidate an otherwise valid primary decision. +- Run deterministic checks for structure, required fields, enums, uniqueness, exact anchors, and protected markers before semantic LLM validation. Semantic validators judge meaning, relevance, quality, rigor, and placement rather than re-deciding machine-verifiable facts. +- Place authoritative role instructions and the response contract before untrusted working context, and keep any production-defined final response-format reminder after assembled context. Retrieved or AI-generated context must never become the terminal instruction. +- Semantic rejection feedback must identify the failed criterion and provide an actionable correction. Structurally complex or repeatedly confused contracts should use focused correct/incorrect examples in production prompt code when needed; simple contracts do not require examples. +- Ordinary compiler outline, construction, review, post-body critique, and semantic-validation contracts are domain-general and exact-objective-directed, using domain- and claim-type-appropriate rigor; mathematics, theorems, proofs, and LaTeX are conditional first-class content, not universal requirements. +- In ordinary compiler modes, `rigor_check` means the submission satisfies the rigor applicable to its domain and claim types. Plain compiler `rigor` remains mathematics-specific; dedicated Lean-verified placement treats Lean as authoritative and limits validation to placement/narrative. The field must not turn ordinary modes into mandatory theorem/proof workflows. +- Tier 3 certainty must preserve evidence status and never promote proposals, hypotheses, unbuilt artifacts, or unperformed experiments into demonstrated results. Format and volume selection follow actual solution dependencies rather than paper count; gap chapters may close proof, mechanism, implementation, evidence, validation, safety, or risk gaps while retaining mathematical/formal-proof support when relevant. -❌ WRONG FORMATS - DO NOT DO THESE: +## Parsing, Retries, and Escaping -1. [Wrong example 1] ❌ NO - [Why it's wrong] -2. [Wrong example 2] ❌ NO - [Why it's wrong] -``` +- Route model-authored JSON through the centralized shared parser/sanitizer. Direct `json.loads()` is reserved for trusted system-written persistence. +- Malformed, truncated, incomplete, scalar, or otherwise contract-incompatible JSON is a parse/contract failure, not a semantic rejection and not content to silently repair into acceptance. The centralized parser may normalize explicitly supported provider compatibility wrappers, such as a non-empty top-level array whose first element is the expected object. Retry remaining contract failures within the role's defined retry path without consuming unrelated workflow attempt counters. +- Never replay raw provider/model transport output or raw response excerpts embedded in parser errors. Retry context may contain only bounded, sanitized visible answer text; exact assistant/tool protocol turns are the narrow exception. +- Keep retries within configured input and output budgets. Never increase user-configured limits to fit retry context; use a simpler bounded repair prompt when needed. +- LaTeX and mathematical notation remain valid JSON string content. Preserve intended decoded text through correct JSON escaping; do not suppress notation, corrupt exact-edit anchors, or double-escape already valid content. -### 2. STRUCTURED REJECTION FEEDBACK -Validator prompts should use this detailed feedback format for complex writing/editing flows; compact validators may use shorter `summary` / `feedback_to_submitter` fields when that is what the code parses: +## Authorized Contract Changes -``` -REJECTION REASON: [Specific Category] - -ISSUE: [What's wrong with the submission] - -WHAT I SAW: -[Exact excerpt or description of the problem] - -WHAT I EXPECTED: -[Concrete example of correct format] - -FIX REQUIRED: -[Step-by-step actionable instructions] -``` - -This format ensures: -- Models understand EXACTLY what was wrong -- Models see CONCRETE examples of correct format -- Feedback is ACTIONABLE, not vague -- Learning from mistakes is possible - -### 3. PRE-VALIDATION CHECKS (RECOMMENDED) -For critical structural requirements (e.g., section headers, required fields), implement **regex-based pre-validation** before LLM validator: - -**Example**: `_pre_validate_outline_structure()` in `compiler_coordinator.py` checks required outline headers (Introduction, Body, Conclusion; optional Abstract when present) using regex patterns before calling the LLM validator. - -**Placeholder Stripping**: Instead of rejecting submissions containing placeholder text, the compiler validator silently strips placeholder markers before validation proceeds. This simplifies the workflow by eliminating rejection feedback loops. - -The stripper removes: -- All exact placeholder constants (`ABSTRACT_PLACEHOLDER`, `INTRO_PLACEHOLDER`, `CONCLUSION_PLACEHOLDER`, `PAPER_ANCHOR`) -- Generic patterns like `[HARD CODED...]` and `[PLACEHOLDER FOR...]` -- Excessive whitespace left behind after stripping - -This approach is more robust than rejection because it handles both intentional reproduction and edge cases where placeholders leak into submissions. - -**Outline vs Paper Confusion Detection**: The `_pre_validate_exact_string_match()` function in `compiler_validator.py` detects when models reference OUTLINE text instead of PAPER text: -- When `old_string` is not found in the paper, the system checks if it exists in the outline -- If found in outline, provides targeted feedback: "OUTLINE_VS_PAPER_CONFUSION" with specific instructions -- This catches the common mistake where models use section headers/text from the outline that haven't been written yet -- Reduces rejection loops by giving models actionable guidance to use CURRENT DOCUMENT PROGRESS - -**Rejection Feedback Flow**: When any pre-validation or LLM validation rejects a submission, the rejection reason is: -1. Logged to `compiler_rejection_log` (included in submitter's "Last 10 rejections" context) -2. Passed directly as `rejection_feedback` to the next submission attempt in the construction loop (displayed prominently with "IMPORTANT - YOUR PREVIOUS RESPONSE WAS REJECTED" header) - -This ensures models learn from mistakes across iterations within the same construction cycle. - -### 4. EXPLICIT VALIDATION CRITERIA -High-risk validator prompts should include explicit criteria with ✓ VALID / ❌ INVALID examples: - -``` -SECTION NAME VALIDATION: - -1. **Abstract Header Check**: - ✓ VALID: A line containing ONLY "Abstract" (case-insensitive) - ❌ INVALID: "Summary of..." or any descriptive text - -2. **Introduction Header Check**: - ✓ VALID: "Introduction" or "I. Introduction" - ❌ INVALID: "Overview" or "Background" -``` - -### 4B. Content Field Validation (Body Phase) -The body construction phase requires that `new_string` is never empty when `needs_construction=true`: - -``` -CONTENT REQUIREMENT - -If you set needs_construction=true, you MUST provide actual content in new_string. -- needs_construction=true + new_string="" is INVALID and will be rejected -- needs_construction=true means you ARE writing content - so PROVIDE it -- Only set needs_construction=false if NO body sections remain to write -- The "new_string" field must NEVER be empty when needs_construction=true - -WRONG (will be rejected): -{ - "needs_construction": true, - "new_string": "", // ❌ INVALID - content is empty but needs_construction is true -} - -CORRECT: -{ - "needs_construction": true, - "new_string": "II. Preliminaries\\n\\nWe begin by establishing...", // ✅ Actual content -} -``` - -### 5. Wrong/Correct Response Patterns -For flows prone to contradictory responses (e.g., phase completion), include explicit examples: - -``` -WRONG RESPONSE (DO NOT DO THIS): -{ - "needs_construction": false, // ❌ Claims no content needed - "section_complete": true, // ❌ Claims phase done - "content": "" // ❌ Provides nothing -} - -CORRECT RESPONSE: -{ - "needs_construction": true, // ✅ Provides content - "section_complete": true, // ✅ Completes phase - "content": "[actual content]" // ✅ Real content -} -``` - -### WHY THESE PRINCIPLES MATTER - -**Without them**: Models experience infinite rejection loops because: -- They don't know what "correct" looks like (ambiguous requirements) -- Validators interpret requirements inconsistently (LLM variance) -- Feedback is vague ("missing X" without showing X) -- Models keep guessing, never learning - -**With them**: Models converge rapidly because: -- Concrete examples eliminate guessing (show don't tell) -- Pre-validation provides consistent structural checks -- Structured feedback is actionable (here's how to fix) -- Models learn patterns from clear before/after examples - -## Key Constraints - -- Keep all existing JSON schemas unchanged -- Keep binary accept/reject validation (no 4-tier system) -- Add "YOUR TASK:" sections with detailed evaluation criteria -- Improve validator rigor (currently lacks evaluation depth) -- Maintain existing prompt assembly order: System → JSON Schema → User Prompt → Context → RAG → Final Instruction -- **MATH VARIANT**: Mathematical theorem/exposition validation focuses on rigor, logical correctness, and established mathematical principles rather than mandatory citation format. Empirical, artifact, and literature claims still require explicit support/citations or conservative wording where compiler validators enforce claim provenance. Models with web search capabilities are encouraged to use them for verification. -- **Proof Candidate JSON Contract**: Automated proof identification is impact-first, not a known-knowledge-base builder. Candidate JSON must use `{"has_provable_theorems": bool, "theorems": [{"theorem_id": str, "statement": str, "formal_sketch": str, "expected_novelty_tier": "major_mathematical_discovery|mathematical_discovery|novel_variant|novel_formulation", "prompt_relevance_rationale": str, "novelty_rationale": str, "why_not_standard_known_result": str}]}`. Every automated proof JSON prompt must treat the USER RESEARCH PROMPT as the primary filter; bounded source-title/brainstorm-topic metadata is context only. Order candidates by direct impact on the user's prompt: direct solutions or impossibility results first, then decisive reductions, obstructions, and structural theorems that themselves make major progress. The three rationale fields must be non-empty or the candidate is skipped before Lean cost. Reject supporting lemmas, routine helpers, standard/textbook/Mathlib restatements, program-local firsts, minor reformulations/local formalizations, trivial/easy proofs, single-tactic/routine proof goals, and general verified background-library entries. Never impose an artificial theorem-count cap unless explicitly requested; user-configurable proof concurrency batching limits simultaneous attempts only and must not truncate identified candidates. -- **Assistant Proof-Support Pack Contract**: Assistant is a separate shared non-blocking proof-support role for the active workflow, not a per-submitter role, proof candidate identifier, or validator. Assistant prompt/output schemas should keep the final model-visible pack capped at 7 fully Lean-verified proof supports by default, each with source/provenance, theorem statement/name, relevance/transfer reason, dependencies/imports, code/hash metadata, and target/freshness metadata. Useful packs are reused for two eligible receiver reads before refresh. Assistant skips true no-external-history targets because it only performs proof-memory retrieval for now. Durable run-scoped cooldown backs off repeated zero-useful retrieval and may shut off Assistant for that run; repeated stagnant same-pack retrieval backs off without shutdown. User live activity should show normal Assistant retrieval result summaries and explicit Assistant model-output failures, not skip/backoff/shutdown turns. Assistant may use current solver feedback/rejections/Lean errors as query material, but outputs must state when support is stale/cached; receivers must independently judge relevance and may ignore any support. Assistant support must not broaden proof-candidate eligibility beyond the user's prompt. Partial/failed artifacts are excluded from the final support pack unless a future explicit obstruction-debug mode says otherwise. -- **Optional `lean_proof` Submission Contract**: Aggregator and LeanOJ brainstorm submitters that choose `submission_type="lean_proof"` should include the same novelty fields (`expected_novelty_tier`, `prompt_relevance_rationale`, `novelty_rationale`, `why_not_standard_known_result`) as ranking context. Submitter-side novelty means public/citable prompt-relevant novelty absent from standard references or Mathlib; program-local firsts do not qualify. The shared Lean proof gate rejects missing/invalid novelty tiers and missing prompt-relevance/novelty/anti-standard-result rationales before Lean cost, in addition to malformed submissions, failed Lean attempts, placeholders, and fake proof devices. Once Lean accepts real proof code, preserve/register the artifact and let novelty/triviality ranking decide context retention, even for not-novel or downshifted actual-theorem artifacts. This preservation rule is not permission to target supporting lemmas. Normal Aggregator validation may reject Lean-verified proof artifacts from the accepted-submission database when the actual theorem is low-impact, trivial, routine, or redundant. LeanOJ brainstorm flow is template-solution-specific: proof-gated `lean_proof` artifacts may be accepted only when useful for exact template obligations. LeanOJ final master-proof editing is template-solution-first and may use standard facts inline when they directly solve current obligations, but it must not accumulate a general known-knowledge library in `master_proof.lean`. -- **Compiler Outline Injection**: The compiler outline is always fully injected (never truncated, never RAGed) for all modes because it provides the structural framework for document construction and validation. -- **TEMPERATURE POLICY**: Default all prompts to `temperature=0.0`. Only two exceptions are allowed: Supercharge candidate attempts and parallel brainstorm submitter lanes. Validators, compiler roles, proof/final roles, and JSON retries must stay deterministic. -- **Supercharge Schema Preservation**: Per-role Supercharge calls generate 4 full answer attempts plus a 5th synthesis answer. Candidate attempts must be sanitized to reusable visible answer text before the 5th call; private thought/channel/control transcript text must never be fed into synthesis, retries, feedback memory, accepted memory, or RAG. The synthesis prompt must place the final instruction after the candidate block, treat candidates as optional working material, and preserve the original task's exact output contract; if the original role expects JSON, the 5th answer must output only valid JSON in that same schema and must not mention Supercharge or candidate attempts. -- **NATURAL COMPLETION POLICY**: Models stop naturally when JSON response is complete. No stop sequences enforced. `sanitize_json_response()` handles trailing whitespace. **CRITICAL**: Truncated JSON (unclosed braces/brackets) raises ValueError - no repair attempted. -- **JSON Response Preprocessing**: All LLM responses preprocessed by `sanitize_json_response()` in `backend/shared/json_parser.py`. See implementation for complete sanitization pipeline: strips reasoning tokens/markdown/control tokens, handles LaTeX escapes (pre-escapes dangerous commands), escapes control chars in strings, rejects truncated JSON, detects pure reasoning text. Enhanced error logging with diagnostics. Array responses auto-extract `data[0]`. -- **Retry Transcript Hygiene**: Raw provider/model transport output must never enter retry prompts, feedback memory, accepted memory, RAG, synthesis prompts, or durable context. Keep conversational retries, but replay only `sanitize_model_output_for_retry_context()` output, which strips known leading private thought/channel/control transcript scaffolding while preserving useful visible malformed JSON/output excerpts and literal tags/operators inside visible content. Channel/control markers must be treated as transport scaffolding only when detected outside visible JSON/string content; sanitization must not treat ordinary Lean/math/operator text such as `<|`, or literal visible text such as ``, ``, `<|channel>final`, or ``, as a control token when it appears inside visible content. Exact assistant/tool protocol turns are the only exception. -- **No Startup Compatibility Testing**: Models trusted to work. JSON sanitizer handles all quirks automatically. Model configs cached on first success. -- **Reasoning Field Extraction**: Agent code checks BOTH `content` and `reasoning` fields for model compatibility. -- **Centralized JSON Parsing**: All agents use `parse_json()` from `backend/shared/json_parser.py`. Exceptions: memory modules loading system-written files use direct `json.loads()`. -- **LeanOJ JSON Retry**: LeanOJ proof-solver roles also use centralized `parse_json()` and must retry malformed/non-object JSON inside the role call before treating the call as failed. Malformed JSON retries do not consume final-attempt cycle counts. Recoverable provider-credit exhaustion is a resumable pause; hard provider/config/privacy/missing-key errors fail visibly with a user-repair path instead of becoming proof feedback. -- **LeanOJ Batch Validation JSON**: LeanOJ brainstorm validation may receive 2-3 submissions and must return `{"decisions": [...]}` with one ordered binary accept/reject decision per submission. Accepted brainstorm decisions should classify `context_role` as `active_plan`, `verified_hint`, `refuted_construction`, or `scratch`; topic validation may receive 2-3 topics and must return ordered `{"decisions": [...]}` entries keyed by `topic_number`. Initial topic validation accepts only broad locked foundation questions that cover `answer n`, lower construction, upper proof, exact LeanOJ semantics, and Lean formalization; reject narrow lemma/tactic/bound/repair topics. -- **LeanOJ Brainstorm Prune JSON**: LeanOJ prune-review prompts must ask whether any accepted brainstorm memory should be removed, updated, or supplemented with one compact corrective idea because it is `outdated`, `redundant`, wrong, harmful, superseded, or missing a needed correction. Do not pressure the reviewer to remove content: keep the conservative `"none"` default, allow at most one operation, and preserve any idea with unique proof-solving value. Prune validation should accept delete/edit/add only when the operation clearly improves the proof-solving database under those criteria. -- **LeanOJ Final Context Routing**: Final-solver direct proof context is limited to verified standalone subproofs plus accepted notes explicitly classified as `active_plan`. Ordinary accepted brainstorm notes default to `scratch`, and accepted idea artifact records must persist `context_role` metadata across resume/reload. Lean-accepted partial scaffolds with `sorry`/`admit` and failed final attempts cannot seed `master_proof.lean` unless explicitly marked high-value/master-seed eligible. The final solver may receive the most recent 5 final attempts only as compact execution feedback to avoid repeating failed edits; this feedback is not proof evidence. Failed/refuted constructions are not proof evidence: pass them only through the compact `refuted_construction_warnings` / “DO NOT USE” channel. -- **LeanOJ Master-Proof Editing JSON**: The final solver edits durable `master_proof.lean` with `{"action":"edit_proof","needs_more_time":true|false,"operation":"full_content|replace|insert_after|delete","old_string":"exact unique proof text","new_string":"Lean code","reasoning":"..."}`. `master_proof.lean` must contain the current chosen proof route only, not accumulated competing/refuted constructions. Final solver prompts must not expose path-transition choices, raw `need_more_brainstorming`, final-cycle failed-attempt counts, or any `stuck_needs_brainstorm` action. They may expose compact recent execution feedback such as Lean errors, stale `old_string` rejections, JSON truncation, and watchdog/no-progress notices. Required corrections from recent feedback must take priority over unrelated new additions, fresh routes, or speculative helpers; new additions are allowed only when they directly implement the required correction or helper code needed for that correction. Phase transitions are selected only by the discrete path-decision mode. Legacy `{"lean_code":...}` is compatibility only. -- **LeanOJ Master-Proof Lean Gate**: A master proof edit must never be persisted merely because the string edit applies. After structural edit application and any required shortening validation, the updated proof is checked in memory first. `needs_more_time=true` edits run Lean with placeholders allowed but still must parse/typecheck, preserve the original template/declarations, and pass forbidden-device integrity checks. `needs_more_time=false` edits run Lean with no placeholders, then final template integrity, answer adequacy, semantic review, and registration. Lean/template failure rejects the edit, preserves the prior master proof and shortening-backup metadata, and feeds the Lean diagnostics (`error_output`, diagnostic output, goal states, raw stderr when present) back to the final solver. -- **LeanOJ Master-Proof Shortening Validation JSON**: Material-shortening edits to `master_proof.lean` must be reviewed before the Lean gate by `leanoj_master_proof_edit_validator` using `{"decision":"accept","reasoning":"...","feedback_to_submitter":""}` or `{"decision":"reject","reasoning":"...","feedback_to_submitter":"precise correction"}`. Rejection preserves the prior proof and becomes direct final-solver feedback. Validator acceptance is not proof acceptance: shortening backup/redo state and `master_proof.lean` persistence happen only after the accepted edit also passes the Lean/template gate. The edit validator must reject changes that ignore required corrections in favor of unrelated new additions, and rejection feedback must instruct the submitter to fix the required corrections before new addition attempts. -- **LeanOJ Final Semantic Review JSON**: After Lean accepts final code and deterministic integrity checks pass, the Final Proof Solver must review the Lean-accepted code against the full LeanOJ problem prompt/template using `{"solved":true,"reasoning":"..."}` or `{"solved":false,"continuation_feedback":"...","reasoning":"..."}`. Rejection is continuation feedback, not verified success. -- **LeanOJ Formalization Semantics Guardrail**: LeanOJ planning, proof-editing, validation, and final-review prompts must state that the Lean template is the formal source of truth, template operations must not be silently reinterpreted to match informal olympiad intuition (e.g. `Nat` subtraction truncates), proposed formulas/constructions should be sanity-checked against the exact Lean predicate on small cases when feasible, and Lean acceptance alone must not be claimed as solving the informal problem unless the formal/informal correspondence is justified. -- **Shared Post-Lean Proof Integrity Gate**: Lean 4 is authoritative for proof checking, but proof outputs still pass `backend/shared/lean_proof_integrity.py` before storage/placement. This shared gate rejects newly introduced `axiom`/`constant`/`opaque` proof devices. Statement-alignment validation classifies mismatches and downshifts storage to the actual Lean-verified theorem instead of discarding real proof artifacts. -- **LeanOJ Proof Validation Boundary**: Lean 4 is authoritative formal checking for LeanOJ success, but LLM validators still gate planning decisions, Lean-accepted subproof relevance, and final semantic review. A compiled subproof must not be stored as verified run context unless it matches the requested subproof/role; a compiled final solution must not stop the run unless it preserves the template and the Final Proof Solver confirms it solves the actual prompt rather than a formal loophole. -- **Aggregator Submitter JSON Retry**: Aggregator submitter retries malformed/non-JSON responses through its standard conversational JSON/LaTeX escaping repair path. The retry preserves sanitized visible failed-output context when useful, but parser exception text inserted into prompts must not replay raw provider output. -- **Standard LaTeX-Focused Retry**: Retry prompts explain HOW to escape LaTeX properly. **LaTeX IS allowed** - just escape backslashes once (`\mathbb` → `\\mathbb`). DO NOT double-escape. For `old_string`: copy EXACTLY from document, just escape backslashes. -- **Retry Context Overflow Prevention (CRITICAL)**: Sanitize failed output, then truncate to ~2000 chars before retry. Parser exception messages that are inserted into retry prompts must report failure type/structure only and must not include raw output excerpts. Calculate if retry fits context window. Fall back to simple re-prompt if too large. Set `max_tokens` explicitly (never `None`). NEVER auto-increase beyond user limits. Applies to: `submitter.py`, `validator.py`, `writer_submitter.py`, `high_param_submitter.py`, `compiler_validator.py`. - -## Internal Content Warning (Required in Most Research/Writing Prompts) - -Most research/writing system prompts include a standardized skepticism warning block. Proof-only helpers such as `proof_prompts.py` and LeanOJ prompts use narrower proof/template guardrails instead of this exact block. This prompt engineering feature prevents AI "echo chambers" where models compound flawed AI-generated content. - -**Why This Exists:** -- All context provided to AI agents (brainstorm databases, accepted submissions, papers, outlines, etc.) is AI-GENERATED within this research system -- This content has NOT been peer-reviewed, published, or externally verified -- Without explicit warnings, models tend to treat internal context as authoritative truth -- This leads to compounding errors where flawed AI-generated claims get built upon - -**Standard Warning Block (included in all prompts):** -``` -INTERNAL CONTENT WARNING - -All context provided to you (brainstorm databases, accepted submissions, papers, reference materials, outlines, previous document content) is AI-generated within this research system. This content has NOT been peer-reviewed, published, or verified by external sources. - -Treat all provided context with extreme skepticism: -- NEVER assume claims are true because they "sound good" or "fit well" -- NEVER trust information simply because it appears in "accepted submissions" or "papers" -- ALWAYS verify information independently before using or building upon it -- NEVER cite internal documents as authoritative or established sources -- Question and validate every assertion, even if it appears in validated content - -WEB SEARCH STRONGLY ENCOURAGED: -If your model has access to real-time web search capabilities (such as Perplexity Sonar or similar), you are STRONGLY ENCOURAGED to use them to: -- Verify mathematical claims against current published research -- Access recent developments and contemporary mathematical literature -- Cross-reference theorems, proofs, and techniques with authoritative sources -- Supplement analysis with verified external information -- Validate approaches against established mathematical consensus - -The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use all available resources - internal context as exploration history, your base knowledge for reasoning, and web search (if available) for verification and current information. - -WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. If you have web search, use it. -``` - -**Files Containing This Warning:** -- `backend/aggregator/prompts/submitter_prompts.py` -- `backend/aggregator/prompts/validator_prompts.py` -- `backend/compiler/prompts/construction_prompts.py` -- `backend/compiler/prompts/outline_prompts.py` -- `backend/compiler/prompts/review_prompts.py` -- `backend/compiler/prompts/rigor_prompts.py` -- `backend/compiler/prompts/critique_prompts.py` -- `backend/compiler/validation/compiler_validator.py` -- `backend/autonomous/prompts/topic_prompts.py` -- `backend/autonomous/prompts/completion_prompts.py` -- `backend/autonomous/prompts/paper_reference_prompts.py` -- `backend/autonomous/prompts/paper_title_prompts.py` -- `backend/autonomous/prompts/paper_redundancy_prompts.py` -- `backend/autonomous/prompts/paper_continuation_prompts.py` -- `backend/autonomous/prompts/final_answer_prompts.py` - -**Note:** The prompt structure examples in the sections below show the core task-specific content. Where used, the INTERNAL CONTENT WARNING block is inserted between the role description and the "YOUR TASK:" section in the actual code. - ---- - -## 1. AGGREGATOR VALIDATOR - -**File:** `backend/aggregator/prompts/validator_prompts.py` - -### Complete Prompt Structure - -**Function:** `get_validator_system_prompt()` - -```python -def get_validator_system_prompt() -> str: - return """You are a validation agent in an AI cluster. Your role is to evaluate mathematical submissions and decide whether they should be added to the shared knowledge base. - -YOUR TASK: -Decide whether the submission provides the strongest rigorous progress currently justified toward solving the user's problem. First prefer avenues that aggressively attack the user's WHOLE question as stated, no partial solutions. - -Essentially, you are evaluating whether the training database becomes more useful toward directly answering the user's mathematical prompt with this submission added than it was without it. - -Note: You are not generating solutions yourself. If the true answer is that the user's question is impossible or has no valid solution as stated, that counts as directly answering the whole question. If a whole-question attack is absolutely not possible in one superintelligence brainstorm, judge whether the submission attacks the next best necessary piece whose resolution would visibly advance the original question. Broader exploratory/background-heavy avenues are valid only when clearly required for that whole-question route. - -META-PHASE EXCEPTION: -If the USER PROMPT explicitly says TOPIC EXPLORATION PHASE or PAPER TITLE EXPLORATION PHASE, evaluate the submission as the requested candidate artifact, not as a direct solution: -- TOPIC EXPLORATION PHASE: accept a candidate brainstorm question if it is specific, distinct, relevant, grounded, and aimed at a strong direct-answer path -- PAPER TITLE EXPLORATION PHASE: accept a candidate title if it is accurate, specific, distinct, professional, and foregrounds direct answer-bearing content when justified -- Do NOT reject these meta-phase submissions merely because they are questions or titles rather than mathematical solutions - -EVALUATION CRITERIA - Consider: -- Does the submission aggressively attack the user's WHOLE question as stated, no partial solutions, or where that is absolutely not possible in one superintelligence brainstorm, the next best necessary piece? -- Does the submission add genuinely new information or perspectives beyond what is already accepted? -- Does the submission connect existing mathematical concepts in novel ways? -- Does the submission provide concrete methods, theorems, proofs, or mathematical techniques? -- Is the submission redundant with current accepted submissions, user provided information, or common mathematical knowledge? -- Is the submission obviously unhelpful or time-wasting content? -- Is the submission grounded in established mathematical principles and rigorous logic? -- Does the submission avoid unfounded claims or logical fallacies? -- Is the submission based on proven mathematical theorems and valid reasoning? - -VALIDATION DECISION RULES: -A submission should be ACCEPTED if it: -1. Aggressively attacks the user's WHOLE question as stated, no partial solutions, OR -2. Addresses the next best necessary piece when a whole-question attack is absolutely not possible in one superintelligence brainstorm, OR -3. Offers rigorous enabling insights only when they materially strengthen a direct route to the full answer and no stronger direct step is available - -A submission should be REJECTED if it: -1. Is redundant with the existing accepted submissions -2. Contains trivial or common mathematical knowledge while also having nothing novel to contribute to the training database -3. Contains logical contradictions or unsupported claims -4. Is too vague or generic to be actionable -5. Is obviously unhelpful or time-wasting content -6. Contains logical fallacies or mathematically unsound reasoning -7. Presents claims as proven without proper mathematical justification - -Ask yourself: "Does adding this submission to our knowledge base make us more capable of solving the user's mathematical prompt than we were without it?" - -Output your decision ONLY as JSON in this exact format: -{ - "decision": "accept or reject", - "reasoning": "Detailed explanation of your decision", - "summary": "Brief summary for feedback, only write this summary if the solution is rejected (max 750 chars)" -}""" -``` - ---- - -## 1A. AGGREGATOR CLEANUP REVIEW - -**File:** `backend/aggregator/prompts/validator_prompts.py` - -**Purpose:** Every 7 coordinator-run-local accepted submissions, the validator performs a cleanup review of the existing database to identify if any previously accepted submission should be removed (due to redundancy, contradictions, or other validation rule violations). Maximum 1 removal per cycle. - -### Complete Prompt Structure - -**Function:** `get_cleanup_review_system_prompt()` - -```python -def get_cleanup_review_system_prompt() -> str: - return """You are a validation agent performing a quality maintenance review of an already-approved knowledge base. - -YOUR TASK: -Review all currently accepted submissions in the knowledge base and determine if ANY ONE submission should be REMOVED because it now violates the original validation criteria. - -Context: -- This is an already-approved database - all submissions passed initial validation -- You are performing a periodic cleanup to maintain database quality -- As the database grows, some submissions may become redundant with newer, better submissions -- You may identify at most one submission for removal (or none) -- It is perfectly acceptable to find no submissions needing removal - -REASONS FOR REMOVAL - A submission should be removed if it: -1. Is now REDUNDANT with other accepted submissions (content is fully covered by other submissions) -2. CONTRADICTS other accepted submissions (logical inconsistencies discovered) -3. Contains information that is now SUPERSEDED by better, more complete submissions -4. Was MARGINALLY useful initially but provides no unique value given the current database state -5. Contains claims that CONFLICT with established mathematical principles evident in other submissions - -REASONS TO KEEP - A submission should be kept if it: -1. Provides unique information that materially strengthens a direct route to the user's full prompt -2. Offers a different perspective or approach that materially improves the best direct solution path -3. Contains specific mathematical details, proofs, or techniques necessary for direct prompt progress -4. Contributes to solution diversity only when that diversity improves credible direct-answer progress - -CONSERVATIVE APPROACH: -- When in doubt, DO NOT recommend removal -- Only recommend removal if you are certain the database would be better without the submission -- A smaller, higher-quality database is better than a large, redundant one - -Selection Rule: -When multiple submissions are redundant with each other, select the weakest one for removal - the one that provides the least unique value. Never remove a more complete submission in favor of keeping a less complete one. - -Output your decision ONLY as JSON in this exact format: -{ - "should_remove": true or false, - "submission_number": number of the submission to remove (or null if should_remove is false), - "reasoning": "Detailed explanation of why this submission should be removed OR why no removal is needed" -}""" -``` - -### JSON Schema - -**Function:** `get_cleanup_review_json_schema()` - -``` -REQUIRED JSON FORMAT: -{ - "should_remove": true OR false, - "submission_number": integer (the submission # to remove) OR null (if should_remove is false), - "reasoning": "string - detailed explanation of your decision" -} -``` - -### Context Handling for Large Databases - -The cleanup review uses `context_allocator.allocate_cleanup_review_context()` which never skips due to size: - -1. **Try direct injection first:** If all submissions fit within context window, direct inject them -2. **Use RAG if too large:** If submissions exceed context window, use RAG retrieval to get relevant chunks -3. **Always proceed:** Cleanup review always runs, using whatever context method is needed - -**Context Allocation Method:** `backend/aggregator/core/context_allocator.py` → `allocate_cleanup_review_context()` - -This method returns: -- `direct`: Direct-injected content (if it fits) -- `rag_context`: RAG-retrieved content (if direct injection was too large) -- `submissions_ragged`: Boolean indicating if RAG was used for submissions - ---- - -## 1B. AGGREGATOR REMOVAL VALIDATION - -**File:** `backend/aggregator/prompts/validator_prompts.py` - -**Purpose:** When a cleanup review proposes a removal, the validator must validate its own removal decision before execution. This is a self-check to prevent erroneous removals. - -**Note:** Removal validation uses RAG when the database is too large for direct injection. It never skips or rejects removal due to prompt size. - -### Complete Prompt Structure - -**Function:** `get_removal_validation_system_prompt()` - -```python -def get_removal_validation_system_prompt() -> str: - return """You are a validation agent reviewing a PROPOSED REMOVAL from the knowledge base. - -YOUR TASK: -A cleanup review has proposed removing a specific submission from the database. You must validate whether this removal should proceed. - -Context: -- Another validation pass has already identified this submission as potentially removable -- You are the final check before the removal is executed -- This is a conservative process - only approve removal if clearly justified - -APPROVE REMOVAL (decision: "accept") if: -1. The submission is genuinely redundant with other submissions -2. The reasoning for removal is sound and well-justified -3. The database would be objectively better without this submission -4. The unique value claimed by the submission is truly covered elsewhere - -REJECT REMOVAL (decision: "reject") if: -1. The submission provides ANY unique value not covered elsewhere -2. The reasoning for removal is weak or unconvincing -3. There is ANY doubt about whether the content is truly redundant -4. Removing would reduce solution diversity or coverage - -CONSERVATIVE DEFAULT: -- If uncertain, REJECT the removal (keep the submission) -- The burden of proof is on REMOVAL, not on keeping - -Output your decision ONLY as JSON in this exact format: -{ - "decision": "accept" or "reject", - "reasoning": "Detailed explanation of why removal should or should not proceed" -}""" -``` - -### JSON Schema - -**Function:** `get_removal_validation_json_schema()` - -``` -REQUIRED JSON FORMAT: -{ - "decision": "accept" OR "reject", - "reasoning": "string - detailed explanation of your validation decision" -} -``` - ---- - -## 1C. AGGREGATOR VALIDATOR (BATCH - 2 SUBMISSIONS) - -**File:** `backend/aggregator/prompts/validator_prompts.py` - -**Purpose:** When the queue has 2 submissions to validate, the validator processes them simultaneously with independent assessment and intra-batch redundancy prevention. - -### Complete Prompt Structure - -**Function:** `get_validator_dual_system_prompt()` - -```python -def get_validator_dual_system_prompt() -> str: - return """You are a validation agent in an AI cluster. Your role is to evaluate TWO mathematical submissions simultaneously and decide whether each should be added to the shared knowledge base. - -YOUR TASK: -Evaluate EACH submission INDEPENDENTLY to determine if it would make a valuable cumulative addition to the shared knowledge base. - -Independent Assessment: -For each submission, ask: "Does this submission provide the strongest rigorous direct progress currently justified toward the user's problem, considering only the existing database (not the other submission in this batch)?" - -Essentially, you are evaluating whether the training database becomes more useful toward directly answering the user's mathematical prompt with each submission added than it was without it. - -META-PHASE EXCEPTION: -If the USER PROMPT explicitly says TOPIC EXPLORATION PHASE or PAPER TITLE EXPLORATION PHASE, evaluate each submission as the requested candidate artifact, not as a direct solution: -- TOPIC EXPLORATION PHASE: accept a candidate brainstorm question if it is specific, distinct, relevant, grounded, and aimed at a strong direct-answer path -- PAPER TITLE EXPLORATION PHASE: accept a candidate title if it is accurate, specific, distinct, professional, and foregrounds direct answer-bearing content when justified -- Do NOT reject these meta-phase submissions merely because they are questions or titles rather than mathematical solutions - -EVALUATION CRITERIA (Apply to EACH submission independently): -- Does the submission add genuinely new information or perspectives beyond what is already accepted? -- Does the submission connect existing mathematical concepts in novel ways? -- Does the submission provide concrete methods, theorems, proofs, or mathematical techniques? -- Is the submission redundant with current accepted submissions, user provided information, or common mathematical knowledge? -- Is the submission obviously unhelpful or time-wasting content? -- Is the submission grounded in established mathematical principles and rigorous logic? -- Does the submission avoid unfounded claims or logical fallacies? - -VALIDATION DECISION RULES (for each submission): -A submission should be ACCEPTED if it: -1. Aggressively attacks the user's WHOLE question as stated, no partial solutions, OR -2. Addresses the next best necessary piece when a whole-question attack is absolutely not possible in one superintelligence brainstorm, OR -3. Offers rigorous enabling insights only when they materially strengthen a direct route to the full answer and no stronger direct step is available - -A submission should be REJECTED if it: -1. Is redundant with the existing accepted submissions -2. Contains trivial or common mathematical knowledge with nothing novel -3. Contains logical contradictions or unsupported claims -4. Is too vague or generic to be actionable -5. Contains logical fallacies or mathematically unsound reasoning - -Intra-Batch Redundancy Prevention: -You must make two separate, independent decisions first - one for each submission. - -STEP 1: Evaluate submission 1 independently against the existing database. Make your accept/reject decision. -STEP 2: Evaluate submission 2 independently against the existing database. Make your accept/reject decision. - -STEP 3: Apply redundancy check only if you independently decided to accept both: -- If you independently decided to accept submission 1 AND independently decided to accept submission 2: - - Check if they cover similar ground or would add redundant information - - If redundant: Keep only the stronger/more complete one, change the weaker to "reject" - - The rejection reason should state: "Redundant with co-submitted submission X which is stronger/more complete" -- If you independently decided to accept only one or reject both: No redundancy check needed - -Note: Each submission gets its own independent decision. Redundancy checking is a tie-breaker between two independently-accepted submissions, not a reason to batch-process decisions. - -DECISION PROCESS (in order): -1. Independently assess submission 1 → accept or reject -2. Independently assess submission 2 → accept or reject -3. If both accepted AND redundant → keep stronger, reject weaker -4. Otherwise → keep your independent decisions - -Output your decisions ONLY as JSON in this exact format: -{ - "decisions": [ - { - "submission_number": 1, - "decision": "accept or reject", - "reasoning": "Detailed explanation of your decision for submission 1", - "summary": "Brief summary for feedback (max 750 chars, only if rejected)" - }, - { - "submission_number": 2, - "decision": "accept or reject", - "reasoning": "Detailed explanation of your decision for submission 2", - "summary": "Brief summary for feedback (max 750 chars, only if rejected)" - } - ] -} -""" -``` - -### JSON Schema - -**Function:** `get_validator_dual_json_schema()` - -``` -REQUIRED JSON FORMAT: -{ - "decisions": [ - { - "submission_number": 1, - "decision": "accept" OR "reject", - "reasoning": "string - detailed explanation of your decision", - "summary": "string - brief summary (max 750 chars, used for rejection feedback)" - }, - { - "submission_number": 2, - "decision": "accept" OR "reject", - "reasoning": "string - detailed explanation of your decision", - "summary": "string - brief summary (max 750 chars, used for rejection feedback)" - } - ] -} - -Example (Accept Both - Non-redundant): -{ - "decisions": [ - { - "submission_number": 1, - "decision": "accept", - "reasoning": "Submission 1 provides a novel approach to modular arithmetic proofs not in existing database.", - "summary": "" - }, - { - "submission_number": 2, - "decision": "accept", - "reasoning": "Submission 2 offers a complementary technique using continued fractions. Not redundant with submission 1.", - "summary": "" - } - ] -} - -Example (Accept One - Redundancy): -{ - "decisions": [ - { - "submission_number": 1, - "decision": "accept", - "reasoning": "Submission 1 provides a comprehensive treatment of the Lindemann-Weierstrass theorem with rigorous proofs.", - "summary": "" - }, - { - "submission_number": 2, - "decision": "reject", - "reasoning": "While independently valuable, submission 2 covers the same Lindemann-Weierstrass material as submission 1 but with less rigor. Accepting only submission 1 to prevent redundancy.", - "summary": "Redundant with co-submitted submission 1 which provides more rigorous coverage." - } - ] -} -``` - ---- - -## 1D. AGGREGATOR VALIDATOR (BATCH - 3 SUBMISSIONS) - -**File:** `backend/aggregator/prompts/validator_prompts.py` - -**Purpose:** When the queue has 3 submissions to validate, the validator processes them simultaneously with independent assessment and all-pairs redundancy checking. - -### Complete Prompt Structure - -**Function:** `get_validator_triple_system_prompt()` - -```python -def get_validator_triple_system_prompt() -> str: - return """You are a validation agent in an AI cluster. Your role is to evaluate THREE mathematical submissions simultaneously and decide whether each should be added to the shared knowledge base. - -YOUR TASK: -Evaluate EACH submission INDEPENDENTLY to determine if it would make a valuable cumulative addition to the shared knowledge base. - -Independent Assessment: -For each of the three submissions, ask: "Does this submission provide the strongest rigorous direct progress currently justified toward the user's problem, considering only the existing database (not the other submissions in this batch)?" - -Essentially, you are evaluating whether the training database becomes more useful toward directly answering the user's mathematical prompt with each submission added than it was without it. - -META-PHASE EXCEPTION: -If the USER PROMPT explicitly says TOPIC EXPLORATION PHASE or PAPER TITLE EXPLORATION PHASE, evaluate each submission as the requested candidate artifact, not as a direct solution: -- TOPIC EXPLORATION PHASE: accept a candidate brainstorm question if it is specific, distinct, relevant, grounded, and aimed at a strong direct-answer path -- PAPER TITLE EXPLORATION PHASE: accept a candidate title if it is accurate, specific, distinct, professional, and foregrounds direct answer-bearing content when justified -- Do NOT reject these meta-phase submissions merely because they are questions or titles rather than mathematical solutions - -EVALUATION CRITERIA (Apply to EACH submission independently): -- Does the submission add genuinely new information or perspectives beyond what is already accepted? -- Does the submission connect existing mathematical concepts in novel ways? -- Does the submission provide concrete methods, theorems, proofs, or mathematical techniques? -- Is the submission redundant with current accepted submissions, user provided information, or common mathematical knowledge? -- Is the submission obviously unhelpful or time-wasting content? -- Is the submission grounded in established mathematical principles and rigorous logic? -- Does the submission avoid unfounded claims or logical fallacies? - -VALIDATION DECISION RULES (for each submission): -A submission should be ACCEPTED if it: -1. Aggressively attacks the user's WHOLE question as stated, no partial solutions, OR -2. Addresses the next best necessary piece when a whole-question attack is absolutely not possible in one superintelligence brainstorm, OR -3. Offers rigorous enabling insights only when they materially strengthen a direct route to the full answer and no stronger direct step is available - -A submission should be REJECTED if it: -1. Is redundant with the existing accepted submissions -2. Contains trivial or common mathematical knowledge with nothing novel -3. Contains logical contradictions or unsupported claims -4. Is too vague or generic to be actionable -5. Contains logical fallacies or mathematically unsound reasoning - -Intra-Batch Redundancy Prevention: -You must make three separate, independent decisions first - one for each submission. - -STEP 1: Evaluate submission 1 independently against the existing database. Make your accept/reject decision. -STEP 2: Evaluate submission 2 independently against the existing database. Make your accept/reject decision. -STEP 3: Evaluate submission 3 independently against the existing database. Make your accept/reject decision. - -STEP 4: Apply redundancy check only among submissions you independently decided to accept: -- If you independently decided to accept multiple submissions (2 or 3): - - Check all pairs for redundancy: (1,2), (1,3), (2,3) - - If any accepted submissions are redundant with each other, keep only the strongest and change the others to "reject" - - Example: If you accepted 1, 2, 3 independently but 1 and 3 cover similar ground: - - Keep 2 (unique) - - Keep whichever of 1 or 3 is stronger - - Change the weaker of 1 or 3 to "reject" with reason: "Redundant with co-submitted submission" -- If you independently decided to accept only one or reject all: No redundancy check needed - -Note: Each submission gets its own independent decision. Redundancy checking is a tie-breaker among independently-accepted submissions, not a reason to batch-process decisions. - -DECISION PROCESS (in order): -1. Independently assess submission 1 → accept or reject -2. Independently assess submission 2 → accept or reject -3. Independently assess submission 3 → accept or reject -4. If multiple accepted AND any pair redundant → keep strongest, reject weaker(s) -5. Otherwise → keep your independent decisions - -FINAL ACCEPTANCE SET MUST BE: -1. Each accepted submission is independently valuable against the existing database -2. NO redundancy exists between ANY accepted submissions -3. The strongest non-redundant combination is chosen - -Output your decisions ONLY as JSON in this exact format: -{ - "decisions": [ - { - "submission_number": 1, - "decision": "accept or reject", - "reasoning": "Detailed explanation of your decision for submission 1", - "summary": "Brief summary for feedback (max 750 chars, only if rejected)" - }, - { - "submission_number": 2, - "decision": "accept or reject", - "reasoning": "Detailed explanation of your decision for submission 2", - "summary": "Brief summary for feedback (max 750 chars, only if rejected)" - }, - { - "submission_number": 3, - "decision": "accept or reject", - "reasoning": "Detailed explanation of your decision for submission 3", - "summary": "Brief summary for feedback (max 750 chars, only if rejected)" - } - ] -} -""" -``` - -### JSON Schema - -**Function:** `get_validator_triple_json_schema()` - -``` -REQUIRED JSON FORMAT: -{ - "decisions": [ - { - "submission_number": 1, - "decision": "accept" OR "reject", - "reasoning": "string - detailed explanation of your decision", - "summary": "string - brief summary (max 750 chars, used for rejection feedback)" - }, - { - "submission_number": 2, - "decision": "accept" OR "reject", - "reasoning": "string - detailed explanation of your decision", - "summary": "string - brief summary (max 750 chars, used for rejection feedback)" - }, - { - "submission_number": 3, - "decision": "accept" OR "reject", - "reasoning": "string - detailed explanation of your decision", - "summary": "string - brief summary (max 750 chars, used for rejection feedback)" - } - ] -} - -Example (Mixed Decisions with Redundancy Handling): -{ - "decisions": [ - { - "submission_number": 1, - "decision": "accept", - "reasoning": "Submission 1 provides a comprehensive proof of the irrationality of sqrt(2) using a novel geometric approach not in existing database.", - "summary": "" - }, - { - "submission_number": 2, - "decision": "reject", - "reasoning": "Submission 2 also addresses sqrt(2) irrationality but uses the standard algebraic proof which is less novel than submission 1's approach. Rejecting to prevent redundancy with submission 1.", - "summary": "Redundant with co-submitted submission 1 which provides a more novel approach." - }, - { - "submission_number": 3, - "decision": "accept", - "reasoning": "Submission 3 explores continued fraction representations - completely different topic from submissions 1 and 2. Adds unique value.", - "summary": "" - } - ] -} - -Example (Reject All): -{ - "decisions": [ - { - "submission_number": 1, - "decision": "reject", - "reasoning": "Submission 1 restates basic definitions already in accepted submission #5.", - "summary": "Redundant with existing submission #5." - }, - { - "submission_number": 2, - "decision": "reject", - "reasoning": "Submission 2 contains vague claims without mathematical rigor.", - "summary": "Too vague and lacks mathematical rigor." - }, - { - "submission_number": 3, - "decision": "reject", - "reasoning": "Submission 3 contains a logical fallacy in its central argument.", - "summary": "Contains logical fallacy - invalid proof structure." - } - ] -} -``` - ---- - -## 2. AGGREGATOR SUBMITTER - -**File:** `backend/aggregator/prompts/submitter_prompts.py` - -### Complete Prompt Structure - -**Function:** `get_submitter_system_prompt()` - -```python -def get_submitter_system_prompt() -> str: - return """You are a mathematical submitter in an AI cluster working to solve complex mathematical problems. Your role is to: - -1. Analyze the user's prompt and provided context carefully -2. Build upon the shared training database (accepted submissions from other agents) -3. Learn from your rejection history to avoid repeating mistakes -4. Generate novel, valuable mathematical insights that advance the solution - -YOUR TASK: -Generate a novel mathematical insight that advances the user's goal. -Generate the strongest rigorous mathematical contribution you can toward the user's goal. Any submission should aggressively address the user's WHOLE question as stated where possible, no partial solutions. - -Focus on mathematical concepts, theorems, techniques, and proofs from your own internal training data that may provide an avenue towards solving or understanding the mathematical problem in the prompt. - -WHAT MAKES A VALUABLE SUBMISSION - Consider: -- Does it add genuinely new information or perspectives beyond what is already in the training database? -- Does it connect existing mathematical concepts in novel ways? -- Does it provide concrete methods, theorems, proofs, or mathematical techniques? -- Is it specific and actionable, not vague or generic? -- Does it increase solution availability or narrow the search space? -- Is it based on established mathematical principles and rigorous logic? - -Content Requirements: -- All submissions must be rooted in sound mathematical reasoning - no unfounded claims or logical fallacies -- Focus on mathematical concepts, theorems, and techniques you know are valid from your training data -- Be specific and actionable, not vague or generic -- Avoid redundancy with existing accepted submissions -- Focus on increasing solution availability or narrowing the search space -- Present rigorous mathematical arguments - -Your submission will be validated against these criteria: -- Does it meaningfully advance the solution space? -- Is it based on sound mathematical principles? -- Does it avoid contradictions? -- Is it non-redundant with existing knowledge? -- Is it mathematically rigorous? - -Output your response ONLY as JSON in this exact format: -{ - "submission": "Your detailed mathematical submission describing concepts, theorems, proofs, and approaches based on established mathematical principles.", - "reasoning": "Brief explanation of why this submission is valuable" -}""" -``` - ---- - -## 3. COMPILER VALIDATOR PROMPTS (SPLIT: OUTLINE VS DOCUMENT) - -**File:** `backend/compiler/validation/compiler_validator.py` - -**Architecture**: The validator uses separate prompt functions for outline validation vs document validation to provide mode-specific criteria and feedback. - -### 3A. OUTLINE VALIDATOR PROMPT - -**Function:** `_get_outline_validation_system_prompt(mode)` (modes: outline_create, outline_update) - -**Anchor Markers**: Outline files use a two-line anchor system: -- Line 1: `[BRACKETED DESIGNATION THAT SHOWS END-OF-PAPER DESIGNATION MARK]` (references document's end marker) -- Line 2: `[HARD CODED END-OF-OUTLINE MARK -- ALL OUTLINE CONTENT SHOULD BE ABOVE THIS LINE]` (outline's own end marker) - -**Required Section Structure**: Every final paper must include these sections. Outlines must include Introduction, Body, and Conclusion; an Abstract heading is allowed but not required because the abstract is written last. - -| Section | Exact Name | Required | Position | -|---------|-----------|----------|----------| -| Abstract | "Abstract" | YES in final paper; optional in outline | First | -| Introduction | "Introduction" or "I. Introduction" | YES | After Abstract | -| Body | Flexible (II., III., etc.) | YES (at least 1) | Between Intro and Conclusion | -| Conclusion | "Conclusion" or "N. Conclusion" | YES | Last required core section; optional appendix/self-review material may follow | - -**Base prompt for OUTLINE modes:** - -```python -base_prompt = """You are validating a mathematical document outline submission. Your role is to decide if this submission should be ACCEPTED or REJECTED. - -REQUIRED SECTION STRUCTURE (MANDATORY): -Every final paper must include these sections; outlines must include the non-abstract sections in this order: -1. **Abstract** - Optional in outlines, named exactly "Abstract" when present; required in the final paper -2. **Introduction** - Must be named exactly "Introduction" or "I. Introduction" (after Abstract if present) -3. **Body Sections** - At least one body section (II, III, IV, etc.) between Introduction and Conclusion -4. **Conclusion** - Must be named exactly "Conclusion" or "N. Conclusion" (always LAST content section) - -OUTLINE VALIDATION CRITERIA: -1. SECTION_STRUCTURE: MUST include Introduction, at least one Body section, and Conclusion with exact names (Abstract is optional but recommended) -2. SECTION_ORDER: [Abstract →] Introduction → Body sections → Conclusion (this exact order, where Abstract is optional) -3. COHERENCE: Logically structured with clear sections and subsections -4. COMPLETENESS: Captures all relevant content from aggregator database in relation to what is relevant to the title set in the user prompt -5. ALIGNMENT: Aligns with user's compiler-directing prompt goals -6. COMPREHENSIVENESS: Provides sufficient detail to guide mathematical document construction -7. MATHEMATICAL PROGRESSION: Body sections follow logical progression (definitions → main results → theorems → proofs) -8. IN-TEXT CITATIONS ONLY: Must NOT include a separate References or Citations section -9. ANCHOR PRESERVATION: Must not attempt to add content after the end-of-outline anchor markers -10. LOGICAL GROUNDING: Outline must reference actual content from aggregator database based on sound mathematical principles, not unfounded claims -11. NO PLACEHOLDER TEXT: Must not contain any placeholder markers (though any that appear will be silently stripped before validation). Placeholders are structural markers that indicate where sections WILL BE written - they should not be intentionally included in submission content. - -YOUR TASK: -Verify the submission meets ALL criteria above. Accept only if ALL criteria pass. Reject if ANY criterion fails. - -REJECTION CATEGORIES (provide specific feedback): -- MISSING_REQUIRED_SECTION: Missing Introduction or Conclusion (must have both with exact names; Abstract is optional) -- INCORRECT_SECTION_ORDER: Sections are out of order (must be: [Abstract →] Introduction → Body → Conclusion, where Abstract is optional) -- INCORRECT_SECTION_NAME: Section names don't match exactly (e.g., "Summary" instead of "Conclusion", "Overview" instead of "Introduction"; if Abstract included, must be "Abstract", "I. Abstract", or "0. Abstract") -- STRUCTURAL: Body sections not in logical mathematical progression order -- INCOMPLETENESS: Missing critical content from aggregator database that's relevant to document title -- MISALIGNMENT: Doesn't serve user's compiler-directing prompt goals -- INSUFFICIENT_DETAIL: Lacks necessary granularity to guide mathematical document construction -- FORMAT_VIOLATION: Includes separate References/Citations section (NOT allowed) -- ANCHOR_VIOLATION: Content placed after outline anchor markers - -Section Name Validation: -- Check for exact presence of "Abstract" (case-insensitive) -- Check for exact presence of "Introduction" (case-insensitive, with or without "I." prefix) -- Check for exact presence of "Conclusion" (case-insensitive, with or without Roman numeral prefix) -- Reject if any of these three required sections is missing or incorrectly named -""" -``` - -**Mode-specific additions:** - -**outline_create mode:** -```python -"""MODE-SPECIFIC CRITERIA (Outline Creation): -- Outline MUST include: Introduction, at least one Body section, Conclusion (Abstract is optional) -- Section names MUST match exactly: "Introduction", "Conclusion" (if Abstract included: "Abstract", "I. Abstract", or "0. Abstract") -- Outline captures all relevant unique content from aggregator database -- Outline provides clear structure for mathematical document construction -- Outline aligns with user's compiler-directing prompt -- Body sections follow logical mathematical progression -- Outline is comprehensive enough to guide entire exposition - -ACCEPT if: All required sections present with correct names + all other criteria met -REJECT if: Missing Introduction/Conclusion, missing body section, incorrect required section names, or any other criterion fails. Abstract is optional in outlines.""" -``` - -**outline_update mode:** -```python -"""MODE-SPECIFIC CRITERIA (Outline Update): -- Update MUST NOT remove or rename Introduction or Conclusion sections; if an Abstract heading is present, it must not be renamed -- Update is necessary (missing content or better structure needed) -- Update follows the existing document's already-constructed format and section ordering -- Update is STRICTLY ADDITIVE ONLY - only adds new sections, never modifies or removes existing structure -- New body sections MUST be inserted between Introduction and Conclusion -- Update respects the current document's construction order -- Update maintains logical progression consistency with existing outline structure -- Update doesn't require modification of already-constructed document content -- Changes are substantive, not cosmetic -- New sections maintain mathematical document ordering -- Update does NOT add a References or Citations section - -Additivity Check: -- Reject if update would rename Introduction or Conclusion, or rename an existing Abstract heading -- Reject if update would insert content after Conclusion (except appendix) -- Reject if update would require editing/moving/renaming already-written document sections -- Reject if update disrupts the flow of existing document content -- Reject if update adds a separate References or Citations section -- Accept only if new outline sections can be added without touching what's already written - -ACCEPT if: All general criteria + mode-specific criteria met AND update is purely additive -REJECT if: Update is unnecessary, harmful, breaks required section structure, or requires document restructuring""" -``` - -### 3B. DOCUMENT VALIDATOR PROMPT - -**Function:** `_get_paper_validation_system_prompt(mode)` (modes: construction, review, rigor, rigor_lean_placement) - -**Base prompt for DOCUMENT modes (construction, review, rigor, rigor_lean_placement):** - -```python -base_prompt = """You are validating a mathematical document construction submission. Your role is to decide if this submission should be ACCEPTED or REJECTED. - -Decision Rule: -If any single criterion below fails, reject the submission. All criteria must pass for acceptance. - -IMPORTANT: Exact string matching (old_string verification) is pre-validated before you see the submission. By the time you receive this validation request, the old_string has already been verified to exist exactly and uniquely in the document. You do NOT need to check if old_string exists - it has been confirmed. Focus on the criteria below. - -DOCUMENT VALIDATION CRITERIA: -1. COHERENCE: Grammatically correct AND maintains holistic document coherence -2. MATHEMATICAL RIGOR: The submission's content is mathematically correct, logically sound, rooted in established mathematical principles (NO unfounded claims or logical fallacies), and is rigorous enough for a mathematical exposition -3. PLACEMENT CONTEXT: Content fits naturally at the specified location (which has been pre-validated), adds imperative content that efficiently contributes towards completion of the document as per the outline, and the content is not overly verbose given the implied scope of the titled document. The content addition adheres to the outline structure for the document. -4. NON-REDUNDANCY: Doesn't repeat existing document content or duplicate any existing sections of the document. The addition strictly adds to the document so the addition does not deviate the document structure from the outline. -5. NO SECTION DUPLICATION: Must not create a section header (e.g., "III. Main Results", "IV. Proofs") OR subsection header (e.g., "V.E.", "IX.C.") that already exists in the CURRENT PAPER/DOCUMENT (actual written content, NOT the outline template). Check BOTH the submission content AND the current document for duplicate section headers and subsection headers. -6. NO FORWARD-LOOKING PREVIEWS: Must not contain forward-looking structural previews (e.g., 'Section II will...', 'We will examine...', organized-as-follows lists) UNLESS this is the introduction/first section where a brief roadmap is allowed -7. IN-TEXT CITATIONS ONLY: Must use in-text citations (if any) and NOT create a separate References or Citations section -8. ANCHOR PRESERVATION: Must not attempt to add content after the end-of-document anchor marker -9. LOGICAL GROUNDING: Must not contain unfounded claims or logical fallacies. All claims must be grounded in established mathematical principles and sound reasoning. -10. NO PLACEHOLDER TEXT: Must not contain any placeholder markers (though any that appear will be silently stripped before validation). Placeholders are structural markers that indicate where sections WILL BE written - they should not be intentionally included in submission content. - -YOUR TASK: -Verify the submission meets ALL criteria above. If even ONE criterion fails, reject the submission. - -Section Duplication Check: -- The current outline is a template showing what sections should be written - it is not actual content -- The current paper/document is the actual written content -- Before accepting, scan only the current paper/document (not the outline) for section headers (e.g., "I.", "II.", "III.", "IV.", "V.") and subsection headers (e.g., "V.E.", "IX.C.", "A.", "B.") -- Check if the submission content contains any section or subsection header that already exists in the current paper/document (actual written content) -- Do not reject just because a section header appears in the outline - the outline is a template, not content -- Reject only if duplicate section headers exist in the actual current paper/document -- Accept if the submission is filling in a section from the outline that hasn't been written yet -- Accept if the submission creates new sections/subsections not present in current document's actual content, or adds content to existing sections without creating duplicate headers - -Pre-Validated Exact String Matching: -- The submission uses operation/old_string/new_string for edit operations -- The old_string has been PRE-VALIDATED to exist exactly and uniquely in the document -- You do NOT need to verify if old_string exists - it has been confirmed by automated pre-validation -- Your role is to assess PLACEMENT CONTEXT: Does the new_string content fit naturally at that location? -- Focus on: logical flow from existing content, appropriate positioning per outline, smooth transitions, no redundancy -- For "replace": Does the new_string improve upon or correctly replace the old_string? -- For "insert_after": Does the new_string flow naturally after the old_string anchor point? -- For "delete": Is removing the old_string appropriate for document quality? -- For "full_content": Does the new section fit the document structure and outline? - -Forward-Looking Preview Detection: -- Reject if content contains: "Section X will...", "organized as follows:", "we will discuss...", "next we examine...", bullet lists describing future sections -- Brief roadmap acceptable only in introduction/first section; reject everywhere else -- Content should be actual mathematical prose (definitions, theorems, proofs, analysis), not structural previews - -Citation Format Check (if applicable): -- Reject if content creates a "References" or "Citations" section header -- Reject if content appears to be a bibliography or reference list -- In-text citations (if present) should be integrated within the narrative - -Mathematical Rigor Check: -- Reject if content contains unfounded claims or logical fallacies -- Accept only content rooted in established mathematical principles and sound reasoning -""" -``` - -**Mode-specific additions (construction example):** - -```python -"construction": """ -MODE-SPECIFIC CRITERIA (Document Construction): -- Outline Adherence: Follows the outline structure appropriately -- Logical Flow: Builds logically from existing document content -- Evidence Integration: Captures relevant aggregator database content -- Non-Repetition: Doesn't repeat existing document sections -- Section Uniqueness: Doesn't create section headers that already exist in current document -- Placement Context: Content fits naturally at the pre-validated location -- Content Check: No forward-looking structural previews outside introduction -- Mathematical Accuracy: Content is based on established mathematical principles and sound reasoning - -VERIFICATION CHECKLIST: -✓ Does it follow the current outline (the outline is the TEMPLATE, not actual content)? -✓ Does it build coherently from what's already written in the ACTUAL DOCUMENT? -✓ Does it integrate content from the aggregator database? -✓ Does it avoid repeating existing content in the ACTUAL DOCUMENT? -✓ Does it avoid creating duplicate section headers that exist in the CURRENT PAPER/DOCUMENT (NOT the outline template)? -✓ Does the new_string content fit naturally at the insertion/replacement location? -✓ Does it avoid forward-looking structural language (unless introduction)? -✓ Is the mathematical content based on established principles and sound reasoning? - -ACCEPT if: All general criteria + all mode-specific criteria met -REJECT if: Any criterion fails (especially duplicate section/subsection headers or unsound mathematical claims) -""" -``` - -**NOTE**: Pre-validation verifies exact string matches before LLM validation. The LLM validator focuses on placement context, not string verification. - ---- - -## 4. WRITING SUBMITTER CONSTRUCTION PROMPTS (PHASE-BASED) - -**File:** `backend/compiler/prompts/construction_prompts.py` - -### Phase-Based Paper Construction Architecture - -Paper construction uses explicit phase-based prompts with `section_complete` feedback. - -**Phase Order (strictly enforced):** -1. **BODY** - All main content sections from the outline -2. **CONCLUSION** - Summary of findings and implications -3. **INTRODUCTION** - Preview of content (written after body so it can accurately describe what follows) -4. **ABSTRACT** - Final summary (signals paper completion) - -### Out-of-Order Writing Rationale (Included in Construction Prompts) - -Each phase-specific construction prompt includes an explanation of WHY papers are written out of order. This explanation is included in the prompts themselves to help the AI understand and follow the correct sequence: - -**Why BODY First:** -- Mathematical content can develop naturally and organically without being constrained by promises made in a pre-written introduction -- If we wrote the introduction or abstract first, it would lock in what the body must contain before we've written it -- The body establishes the actual mathematical content with full flexibility - -**Why CONCLUSION Second (before Introduction):** -- Writing the conclusion now, before the introduction, ensures we summarize actual proven results rather than hypothetical content -- The conclusion summarizes what was ACTUALLY written in the body sections - -**Why INTRODUCTION Third:** -- Written AFTER body and conclusion, so it can accurately describe what the paper actually contains -- The introduction will accurately describe both the body content AND the conclusion -- Prevents forward-looking language that refers to content not yet written - -**Why ABSTRACT Last:** -- Written as the final section to summarize the COMPLETE paper -- Can only be accurate after all other content exists -- Signals paper completion when validated - -**Prompt Content:** -The actual prompt text included in each phase-specific construction prompt includes: -``` -IMPORTANT - WHY WE WRITE PAPERS OUT OF ORDER: -This paper is constructed OUT OF ORDER intentionally. The writing sequence is: -1. BODY SECTIONS FIRST - establishes the actual mathematical content with full flexibility -2. CONCLUSION second - summarizes what was actually proven in the body -3. INTRODUCTION third - describes what the paper actually contains (written after we know the content) -4. ABSTRACT last - summarizes the complete paper -``` - -**PHASE TRANSITION MECHANISM:** -- Each phase uses a dedicated prompt function -- Submitter explicitly sets `section_complete: true` when current phase is finished -- Coordinator advances to next phase ONLY on explicit `section_complete` signal AND validates section exists -- **Content validation enforced**: Before transitioning phases, coordinator verifies the required section header exists in paper using regex patterns (not just placeholders) -- Prevents premature phase transitions when content wasn't actually written -- Replaces unreliable regex-based detection of section headers - -### Section Placeholder System - -**Purpose:** Placeholders make it crystal clear to the AI where sections will be written and that they do NOT exist yet. This fixes the problem of models falsely claiming sections like Introduction are "already written" when they are not. - -**How It Works:** -- When the first body section is accepted, the paper is initialized with placeholder markers for Conclusion, Introduction, and Abstract -- Each placeholder clearly states what section goes there and when it will be written -- When a non-body section (Conclusion/Introduction/Abstract) is validated and accepted, the corresponding placeholder is REPLACED with the actual content -- Placeholders are visible to the AI in CURRENT DOCUMENT PROGRESS - the AI sees them explicitly - -**Placeholder Constants (defined in `backend/compiler/memory/paper_memory.py`):** -``` -ABSTRACT_PLACEHOLDER = "[HARD CODED PLACEHOLDER FOR THE ABSTRACT SECTION - TO BE WRITTEN AFTER THE INTRODUCTION IS COMPLETE]" -INTRO_PLACEHOLDER = "[HARD CODED PLACEHOLDER FOR INTRODUCTION SECTION - TO BE WRITTEN AFTER THE CONCLUSION SECTION IS COMPLETE]" -CONCLUSION_PLACEHOLDER = "[HARD CODED PLACEHOLDER FOR THE CONCLUSION SECTION - TO BE WRITTEN AFTER THE BODY SECTION IS COMPLETE]" -``` - -**Note**: These placeholders are structural markers only. Submissions must never contain placeholder text. - -### System-Managed Markers Explanation (Included in Prompts) - -Construction and outline prompts include detailed explanations telling the AI about system-managed markers. This explanation is important because: -- The AI sees placeholders in CURRENT DOCUMENT PROGRESS but did NOT create them -- Without explanation, the AI might try to output these markers or misunderstand their purpose -- The explanation prevents confusion about what exists vs. what doesn't - -**Prompt Content Explaining System-Managed Markers:** -``` -SYSTEM-MANAGED MARKERS (NOT YOUR OUTPUT): - -The paper uses placeholder markers that the system adds automatically (you did not create these): - -**Section Placeholders** (show where sections will be written): -- [HARD CODED PLACEHOLDER FOR THE ABSTRACT SECTION...] - Will be replaced when Abstract is written -- [HARD CODED PLACEHOLDER FOR INTRODUCTION SECTION...] - Will be replaced when Introduction is written -- [HARD CODED PLACEHOLDER FOR THE CONCLUSION SECTION...] - Will be replaced when Conclusion is written - -**Paper Anchor** (marks document boundary): -- [HARD CODED END-OF-PAPER MARK -- ALL CONTENT SHOULD BE ABOVE THIS LINE] - -Distinctions: -1. **Placeholders/anchors in CURRENT DOCUMENT PROGRESS**: These are SYSTEM-MANAGED. The code in paper_memory.py adds them automatically. You did NOT create them. - -2. **You must NEVER output these markers in YOUR SUBMISSIONS**: While the system will silently strip any placeholder markers that accidentally appear, they should never be included in your submissions. Focus on writing actual mathematical content. - -HOW PLACEHOLDERS WORK: -- When you see a placeholder in CURRENT DOCUMENT PROGRESS, that section has NOT been written yet -- When you write that section and it's validated, the system will REPLACE the placeholder with your content -- If you see actual section content (not a placeholder), that section IS already written -- Placeholders make it crystal clear what exists vs. what doesn't - -WHY THEY EXIST: They prevent AI confusion about whether sections like Introduction are "already written" when they aren't. -``` - -**Placeholder Stripping:** -Submissions are automatically scanned for placeholder text before validation. Any placeholder markers found (e.g., `[HARD CODED PLACEHOLDER`, `[HARD CODED END-OF`) are silently stripped. This prevents placeholder text from interfering with validation while avoiding unnecessary rejection loops. - -**Initial Paper Structure (after first body section accepted):** -``` -[PLACEHOLDER FOR THE ABSTRACT SECTION - TO BE WRITTEN AFTER THE INTRODUCTION IS COMPLETE] - -[PLACEHOLDER FOR INTRODUCTION SECTION - TO BE WRITTEN AFTER THE CONCLUSION SECTION IS COMPLETE] - -... body content ... - -[PLACEHOLDER FOR THE CONCLUSION SECTION - TO BE WRITTEN AFTER THE BODY SECTION IS COMPLETE] - -[HARD CODED END-OF-PAPER MARK -- ALL CONTENT SHOULD BE ABOVE THIS LINE] -``` - -**Placeholder Replacement Rules:** -1. When Conclusion is accepted → `CONCLUSION_PLACEHOLDER` is replaced with actual Conclusion content -2. When Introduction is accepted → `INTRO_PLACEHOLDER` is replaced with actual Introduction content -3. When Abstract is accepted → `ABSTRACT_PLACEHOLDER` is replaced with actual Abstract content -4. Each placeholder is replaced exactly ONCE with validated content -5. Placeholders are replaced via exact string matching (old_string = placeholder, new_string = content) - -**Why This Fixes the "Section Already Exists" Bug:** -- Previously, the AI would see body content + end-of-paper anchor and incorrectly conclude "the paper is complete" -- Now the AI explicitly sees `[PLACEHOLDER FOR INTRODUCTION SECTION...]` which makes it impossible to claim the Introduction exists when it doesn't -- The prompt instructions tell the AI: "If you see a placeholder, that section has NOT been written yet" - -**Placeholder Boundary Invariant:** -The CONCLUSION_PLACEHOLDER acts as a hard boundary that body content cannot cross: -- When `_apply_edit()` handles `insert_after` operations for body content: - 1. Exact string matching finds the specified old_string - 2. Checks if the insertion point would be after CONCLUSION_PLACEHOLDER - 3. **AUTO-CORRECTION**: If the anchor is after the placeholder, automatically relocates insertion to just BEFORE the placeholder (prevents infinite rejection loops) - 4. Inserts new_string at the correct position, ensuring content stays before the placeholder - 5. Placeholders are treated as exact strings that can be matched and replaced -- This ensures body sections always accumulate above the conclusion marker -- The boundary is enforced in `compiler_coordinator.py` `_apply_edit()` method -- Works identically for both manual (Part 2) and autonomous (Part 3) modes - -**Placeholder Resume Repair (Critical for Crash Recovery):** -When the compiler resumes from an existing paper file, it checks if placeholders exist and adds them if missing: -- **Problem**: Papers created before the placeholder system or from older code versions may not have placeholders -- **Symptom**: "old_string not found in document" pre-validation failures when the model tries to use placeholder text as old_string -- **Solution**: `paper_memory.ensure_placeholders_exist()` is called when resuming from an existing paper -- **Behavior**: - 1. Checks if ABSTRACT_PLACEHOLDER, INTRO_PLACEHOLDER, CONCLUSION_PLACEHOLDER, and PAPER_ANCHOR exist - 2. If any are missing, extracts the body content and reconstructs the paper with all placeholders in correct positions - 3. Logs: "Placeholders were missing and have been added to the paper" -- **Implementation**: Called in `compiler_coordinator._main_workflow()` when `is_resuming_paper=True` -- **Files**: `backend/compiler/memory/paper_memory.py` (ensure_placeholders_exist method), `backend/compiler/core/compiler_coordinator.py` (resume logic) - -**Fake Placeholder Detection (Bug Fix 2026-01-22):** -Models sometimes insert fake placeholder text during body construction (e.g., "XI. Conclusion\n*placeholder will be replaced...*"). Detection checks FULL content length (not sample): >300 chars = real; <300 chars with keywords ("will be replaced", "placeholder") = fake; <300 chars no keywords = real if >50 chars. Prevents treating fake placeholders as written sections. - -### Phase-Specific Prompt Functions - -**Body Phase:** `get_body_construction_system_prompt()` -- Writes all main content sections following the outline -- Sets `section_complete: true` when ALL body sections are written -- Then transitions to CONCLUSION phase - -**Conclusion Phase:** `get_conclusion_construction_system_prompt()` -- Must write the conclusion section content with `needs_construction: true` -- Cannot respond with `needs_construction: false` - this phase requires writing content -- Summarizes findings from completed body sections -- Must set both `needs_construction: true` and `section_complete: true` when submitting conclusion content -- Writing the conclusion completes the conclusion phase (single submission) -- Then transitions to introduction phase - -**Introduction Phase:** `get_introduction_construction_system_prompt()` -- Must write the introduction section content with `needs_construction: true` -- Cannot respond with `needs_construction: false` - this phase requires writing content -- Writes the introduction with roadmap (now knows actual content from body + conclusion) -- Must set both `needs_construction: true` and `section_complete: true` when submitting introduction content -- Writing the introduction completes the introduction phase (single submission) -- Then transitions to abstract phase - -**Abstract Phase:** `get_abstract_construction_system_prompt()` -- Must write the abstract section content with `needs_construction: true` -- Cannot respond with `needs_construction: false` - this phase requires writing content -- Writes the abstract as final summary of complete paper -- Must always set both `needs_construction: true` and `section_complete: true` when submitting abstract content -- Writing abstract completes the paper - there is no next step after abstract (this is the final phase) -- Triggers paper completion workflow - -### JSON Schema (All Construction Phases) - -``` -REQUIRED JSON FORMAT: -{ - "needs_construction": true or false, - "content": "Your complete section/paragraph text (empty if needs_construction=false)", - "operation": "replace | insert_after | delete | full_content", - "old_string": "Exact text to find in document (empty for full_content or needs_construction=false)", - "new_string": "Replacement/insertion text (empty if needs_construction=false)", - "section_complete": true or false, - "reasoning": "Why construction is or isn't needed, and section completion status" -} - -section_complete field: -- Set to true only when the entire current phase is complete -- For body phase: true when all body sections from outline are written -- For conclusion phase: Must set needs_construction=true and provide content when marking complete -- For introduction phase: Must set needs_construction=true and provide content when marking complete -- For abstract phase: Must always set needs_construction=true and provide content (this is the final phase) -- Setting section_complete=true triggers transition to next phase - -needs_construction requirements by phase: -- **Body**: Can be false if all body sections written (no more content needed before conclusion) -- **Conclusion**: Must be true - cannot complete this phase without writing the conclusion -- **Introduction**: Must be true - cannot complete this phase without writing the introduction -- **Abstract**: Must be true - cannot complete this phase without writing the abstract - -Invalid pattern (prohibited): -needs_construction=false + section_complete=true + content="" = invalid for conclusion/intro/abstract phases - -Required pattern: -needs_construction=true + section_complete=true + content="..." = required for conclusion/intro/abstract phases -``` - -### Example JSON Outputs - -**Body Phase - Still Working:** -```json -{ - "needs_construction": true, - "content": "III. Main Results\n\nWe now present our central findings...", - "operation": "insert_after", - "old_string": "This completes the preliminaries. ∎", - "new_string": "III. Main Results\n\nWe now present our central findings...", - "section_complete": false, - "reasoning": "Writing Section III Main Results. More body sections remain from outline." -} -``` - -**Body Phase - Complete:** -```json -{ - "needs_construction": false, - "content": "", - "operation": "", - "old_string": "", - "new_string": "", - "section_complete": true, - "reasoning": "All body sections from the outline have been written. Ready to write Conclusion." -} -``` - -**Conclusion Phase - Complete:** -```json -{ - "needs_construction": true, - "content": "VI. Conclusion\n\nIn this paper, we have established...", - "operation": "replace", - "old_string": "[HARD CODED PLACEHOLDER FOR THE CONCLUSION SECTION - TO BE WRITTEN AFTER THE BODY SECTION IS COMPLETE]", - "new_string": "VI. Conclusion\n\nIn this paper, we have established...", - "section_complete": true, - "reasoning": "Conclusion section written. This completes the conclusion phase." -} -``` - -**Abstract Phase - Paper Complete:** -```json -{ - "needs_construction": true, - "content": "Abstract\n\nThis paper presents a rigorous analysis of...", - "operation": "replace", - "old_string": "[HARD CODED PLACEHOLDER FOR THE ABSTRACT SECTION - TO BE WRITTEN AFTER THE INTRODUCTION IS COMPLETE]", - "new_string": "Abstract\n\nThis paper presents a rigorous analysis of...", - "section_complete": true, - "reasoning": "Abstract written. Paper is now complete." -} -``` - -**Paper Context Injection Rules:** -| Phase | Paper Included? | Reason | -|-------|----------------|--------| -| BODY (first submission) | YES (shows as EMPTY) | Must see paper is empty to use full_content operation | -| BODY (continuation) | YES | Must see existing sections to continue coherently | -| CONCLUSION | YES | **MUST** see body sections to summarize them | -| INTRODUCTION | YES | **MUST** see body+conclusion to preview them | -| ABSTRACT | YES | **MUST** see entire paper to summarize it | - -**Implementation**: Paper state is **ALWAYS shown** in all phases. When empty, displays "(EMPTY - no content written yet)" so model knows to use `operation='full_content'`. This prevents models from confusing outline structure with paper content and using invalid `insert_after` operations on empty documents. - -### Legacy Generic Prompt (Backward Compatibility) - -**Function:** `get_construction_system_prompt()` - Still available for non-phase-based usage - -When `section_phase` is not specified, the generic prompt is used which follows the old behavior of writing "next best section" without explicit phase constraints. - -### Coordinator Phase Tracking - -**File:** `backend/compiler/core/compiler_coordinator.py` - -The coordinator tracks the current phase via `autonomous_section_phase`: -- Starts at "body" when paper construction begins -- Advances phases based on `section_complete` signal from submissions -- Phase order: body → conclusion → introduction → pre-abstract empirical red-team review → abstract -- Paper is complete when abstract phase receives `section_complete: true` - -**Phase Transition Method:** `_check_phase_transition(section_complete: bool)` -- Takes explicit `section_complete` parameter from submission -- Returns `True` when paper is fully complete (abstract phase done) -- Broadcasts WebSocket events for each phase transition - -**Autonomous Mode Phase Synchronization:** -When the compiler runs in autonomous mode (Part 3), the autonomous coordinator polls `autonomous_section_phase` every 3 seconds and syncs it to the workflow state (`paper_phase` field). This ensures accurate crash recovery and prevents the bug where workflow state showed "outline" even though the compiler had progressed through body/conclusion/introduction phases. - -### Retroactive Brainstorm Operation (Optional, Autonomous Mode Only) - -During autonomous paper compilation, the construction JSON includes an optional `brainstorm_operation` field: - -```json -{ - ... (standard construction fields) ..., - "brainstorm_operation": { - "action": "edit | delete | add", - "submission_number": 5, - "new_content": "corrected or new content (empty for delete)", - "reasoning": "Independent justification" - } -} -``` - -**Validation**: Brainstorm operations are validated by the compiler validator with brainstorm-only context. The validator never sees the paper operation when validating a brainstorm operation. - -**Independent Validity Principle**: Each operation must be justified on its own merits. Paper content must not depend on a brainstorm correction. Brainstorm corrections must not depend on paper content. - -**Models**: `BrainstormRetroactiveOperation` in `models.py`. Parsed in `writer_submitter.py`. Handled in `compiler_coordinator._handle_brainstorm_retroactive_operation()`. Validated in `compiler_validator.validate_brainstorm_operation()`. - ---- - -## 5. COMPILER-Submitter OUTLINE CREATION (PHASE 1: ITERATIVE REFINEMENT) - -**File:** `backend/compiler/prompts/outline_prompts.py` - -### Two-Phase Outline System - -**PHASE 1: Initial Outline Creation (Before Paper Construction)** -- Iterative refinement loop with validator feedback -- Submitter generates/refines outline multiple times -- Validator provides feedback each iteration (accept/reject) -- **CRITICAL: When accepted, feedback includes the outline content** so model can see its own work -- Submitter decides when to lock outline (`outline_complete=true`) -- Hard limit: 15 iterations (force completion) - -**PHASE 2: Outline Update (During Body Construction) - UPDATED FOR EXACT STRING MATCHING** -- Runs once per construction loop cycle (after 4 constructions) -- Construction loop repeats throughout body construction -- If rejected: moves to review phase, gets another chance next loop -- Only during body phase (skipped for conclusion/intro/abstract) -- Uses `needs_update`, `operation`, `old_string`, `new_string` JSON schema -- Now uses exact string matching like all other compiler modes - -### Required Section Structure (MANDATORY) - -Final papers must include these sections. Outlines must include Introduction, Body, and Conclusion; an Abstract heading is allowed but not required because the abstract is written last. - -| Section | Exact Name | Required | Position | -|---------|-----------|----------|----------| -| Abstract | "Abstract" | YES in final paper; optional in outline | First | -| Introduction | "Introduction" or "I. Introduction" | YES | After Abstract | -| Body | Flexible (II., III., etc.) | YES (at least 1) | Between Intro and Conclusion | -| Conclusion | "Conclusion" or "N. Conclusion" | YES | Last required core section; optional appendix/self-review material may follow | - -### JSON Schema - Phase 1 (Outline Create) - -```json -{ - "content": "string - complete outline with sections and subsections", - "outline_complete": true OR false, - "reasoning": "string - explanation of outline structure AND completion decision" -} -``` - -**⚠️ CRITICAL - OUTLINE CREATE MODE USES ONLY THESE 3 FIELDS:** -- `content` - Your complete outline text; it must include Introduction, at least one Body section, and Conclusion. Abstract may be included first but is not required. -- `outline_complete` - true or false -- `reasoning` - Your explanation - -**❌ DO NOT USE THESE FIELDS IN OUTLINE CREATE MODE:** -- `operation` - WRONG (this is for outline_update mode ONLY) -- `old_string` - WRONG (this is for outline_update mode ONLY) -- `new_string` - WRONG (this is for outline_update mode ONLY) -- `needs_update` - WRONG (this is for outline_update mode ONLY) - -### Examples - Phase 1 - -**Iteration 2 (Continue Refining) - ✅ CORRECT:** -```json -{ - "content": "Abstract\n\nI. Introduction\n A. Historical context\n B. Problem statement\n\nII. Preliminaries\n A. Basic definitions\n B. Constructible numbers\n C. Transcendental numbers\n\nIII. Main Results\n A. Theorem 1\n B. Theorem 2\n\nIV. Conclusion", - "outline_complete": false, - "reasoning": "Structure improved based on validator feedback. Added Problem statement and Transcendental numbers subsections. However, validator feedback noted missing Galois theory connections (submission 11) and incomplete coverage of Lindemann-Weierstrass theorem details. Will add these refinements in next iteration." -} -``` - -**Note:** The example includes an optional Abstract heading; this is allowed but not mandatory for outlines. - -**❌ WRONG - DO NOT DO THIS:** -```json -{ - "operation": "full_content", - "new_string": "Abstract\n\nI. Introduction\n...", - "content": "I. Introduction\n...", - "outline_complete": false -} -``` - -**This is WRONG because:** -1. Uses `operation` field (only for outline_update mode, NOT outline_create) -2. Uses `new_string` field (only for outline_update mode, NOT outline_create) -3. `content` field omits the required Introduction/Body/Conclusion outline structure - -**Iteration 7 (Lock Outline):** -```json -{ - "content": "Abstract\n\nI. Introduction\n A. Historical context of circle-squaring problem\n B. Statement of impossibility\n C. Overview of proof approach\n\nII. Preliminaries and Definitions\n A. Compass and straightedge constructions\n B. Field extensions and constructible numbers\n C. Algebraic vs. transcendental numbers\n D. Galois theory connections to constructibility\n\nIII. Main Theoretical Results\n A. Theorem: Characterization of constructible lengths\n B. Theorem: Lindemann-Weierstrass (transcendence of π)\n C. Theorem: Baker's theorem and applications\n D. Corollary: Impossibility of squaring the circle\n\nIV. Proofs and Derivations\n A. Proof of constructibility characterization\n B. Lindemann-Weierstrass proof outline\n C. Connection to algebraic field theory\n D. Derivation of main impossibility result\n\nV. Conclusion\n A. Summary of impossibility result\n B. Related classical problems\n C. Historical and mathematical significance", - "outline_complete": true, - "reasoning": "Outline comprehensively captures ALL content from aggregator database. Added Galois theory subsection D under Preliminaries (iterations 3-4 feedback). Expanded Main Results with Baker's theorem (iteration 5 feedback). Added algebraic field theory connection in Proofs section (iteration 6 feedback). All required sections (Introduction, Body, Conclusion) present with exact names, plus optional Abstract section. Structure follows logical mathematical progression. Ready to lock and begin paper construction." -} -``` - -### Validator Feedback Format - -**Acceptance with Optional Improvements:** -``` -ITERATION 4 - ACCEPTED - -STRENGTHS: -- All required sections (Introduction, Body sections, Conclusion) present with correct names, plus optional Abstract -- Good coverage of Lindemann-Weierstrass theorem and constructibility theory -- Logical progression from basic definitions to advanced theorems - -VALIDATION CRITERIA MET: -✓ Required section structure (Intro, Body, Conclusion) with optional Abstract -✓ Captures content from submissions 1, 3, 5, 7, 9, 12, 14 -✓ Aligns with user's compiler-directing prompt goal -✓ Sections in correct order with valid names - -OPTIONAL IMPROVEMENTS TO CONSIDER: -- Galois theory connections mentioned in submission 11 could strengthen Preliminaries section -- Consider adding subsection on worked examples under Proofs for pedagogical clarity -- Baker's theorem (submissions 8, 12, 17) currently not reflected in outline - -DECISION: You may refine further (outline_complete=false) or lock now (outline_complete=true). The outline is acceptable as-is, but the optional improvements would enhance comprehensiveness. -``` - -**Rejection with Specific Guidance:** -``` -ITERATION 2 - REJECTED - -VALIDATION FAILURE: MISSING REQUIRED SECTION -The outline is missing the "Conclusion" section. Every outline MUST include: Introduction, at least one Body section, and Conclusion with exact names (Abstract is optional). - -ADDITIONAL CONTENT GAPS: -- Section II "Preliminaries" lacks coverage of Galois theory connections to constructibility (extensively discussed in submissions 8, 11, 15) -- Section III "Main Results" missing Baker's theorem, which appears prominently in submissions 8, 12, 17, 20 -- No coverage of algebraic field theory foundations from submissions 6, 13, 18 - -REQUIRED ACTIONS: -1. Add "Conclusion" section as the last content section (required) -2. Add subsection "D. Galois Theory Connections" under Section II -3. Add subsection "C. Baker's Theorem" under Section III -4. Consider adding subsection on algebraic field theory under Preliminaries - -STRUCTURAL NOTE: -Current body sections (II. Preliminaries, III. Main Results) are well-structured. Once you add Conclusion and fill content gaps, the outline will meet validation criteria. - -Generate revised outline addressing these specific gaps. -``` - ---- - -## 6. COMPILER-Submitter OUTLINE UPDATE - -**File:** `backend/compiler/prompts/outline_prompts.py` - -### Complete Prompt Structure - -**Function:** `get_outline_update_system_prompt()` - -```python -def get_outline_update_system_prompt() -> str: - return """You are reviewing the current document outline to decide if it needs updating. Your role is to: - -1. Review the aggregator database for any content not yet captured in the outline -2. Review the current document construction progress -3. Decide if the outline needs modification to better serve the document - -REQUIRED SECTION STRUCTURE (MUST BE PRESERVED): -The outline MUST maintain these exact sections in this exact order: -1. **Abstract** - Brief summary (exactly "Abstract") -2. **Introduction** - Background and roadmap (exactly "Introduction" or "I. Introduction") -3. **Body Sections** - Main content (numbered II, III, IV, etc.) -4. **Conclusion** - Summary of findings (exactly "Conclusion" or "N. Conclusion") - -YOUR TASK: -Decide if the outline requires updates. Consider: -- Relevance to current content from aggregator database -- Missing content that should be included in outline -- Structural issues in current outline -- Alignment with document construction progress - -WHEN TO UPDATE THE OUTLINE (ADDITIONS ONLY): -- Important content from aggregator DB is missing from current outline -- Document construction reveals needed additional sections -- New pertinent information should be added to complete the document relative to the user-prompt title - -Note: You can only add to the outline, not delete or remove existing sections. -Note: New body sections must be inserted before the Conclusion section. - -WHEN NOT TO UPDATE: -- Current outline already contains all pertinent information -- No new relevant content needs to be added - -Requirements for Updates: -- All content must be rooted in sound mathematical reasoning from the aggregator database -- No unfounded claims or logical fallacies -- Focus on rigorous mathematical arguments -- Never change the names of Abstract, Introduction, or Conclusion sections -- New body sections must be inserted between Introduction and Conclusion - -EXACT STRING MATCHING FOR UPDATES: -This system uses EXACT STRING MATCHING. To insert new sections: -1. Find the exact text in the outline where you want to insert AFTER (the anchor) -2. Use operation="insert_after" with that exact anchor text as old_string -3. Put your new section(s) as new_string - -If you decide NO update is needed, set "needs_update" to false and leave operation, old_string, and new_string empty. - -Output your response ONLY as JSON in this exact format: -{ - "needs_update": true or false, - "operation": "insert_after | replace", - "old_string": "exact text from outline (anchor point, empty if needs_update=false)", - "new_string": "new sections to add (empty if needs_update=false)", - "reasoning": "Why addition is or isn't needed" -} -""" -``` - ---- - -## 7. COMPILER SUBMITTER REVIEW/AUDIT DOCUMENT PROMPT - -**File:** `backend/compiler/prompts/review_prompts.py` - -### Complete Prompt Structure - -**Function:** `get_review_system_prompt()` - -```python -def get_review_system_prompt() -> str: - return """You are reviewing the current mathematical document draft for errors and needed improvements. Your role is to: - -1. Review ONLY the current document (aggregator database is NOT in your context for this task) -2. Identify any obvious errors or issues -3. Decide if an edit is needed or if the document is acceptable as a "draft in progress" - -YOUR TASK: -Review the document for these specific issues: -- Grammar errors -- Clarity problems -- Mathematical accuracy issues -- Logical errors or gaps -- Structural issues -- Redundancy -- Forward-looking structural previews -- Other improvements - -WHEN TO MAKE AN EDIT: -- Clear grammatical errors -- Obvious redundancy that should be removed -- Coherence issues between sections -- Terminology inconsistencies -- Mathematical inaccuracies or logical errors -- Significant clarity improvements possible -- Forward-looking structural language outside introduction (e.g., 'Section III will...', bulleted lists of future content) -- Unfounded claims or logical fallacies that should be corrected - -WHEN NOT TO MAKE AN EDIT: -- Document is acceptable for a draft in progress -- Only minor stylistic preferences -- Changes would be purely cosmetic -- No obvious issues found - -EXACT STRING MATCHING FOR EDITS (PRE-VALIDATED): -This system uses EXACT STRING MATCHING with automated pre-validation: -1. Find the EXACT text you want to modify (must exist verbatim in document) -2. Choose the operation: "replace", "insert_after", or "delete" -3. Specify old_string (exact match) and new_string (replacement/insertion) -4. Pre-validation will automatically verify the old_string exists and is unique before LLM validation - -If NO edit is needed, set "needs_edit" to false and leave operation, old_string, and new_string empty. - -Output your response ONLY as JSON in this exact format: -{ - "needs_edit": true or false, - "operation": "replace | insert_after | delete", - "old_string": "exact text to find (must exist verbatim, empty if needs_edit=false)", - "new_string": "replacement text (empty for delete or if needs_edit=false)", - "content": "full content for logging", - "reasoning": "Why edit is or isn't needed" -} -""" -``` - ---- - -## 8. CRITIQUE & SELF-REVIEW PHASE (POST-BODY CONSTRUCTION) - -**Owner:** Rigor & Proofs Submitter generation (`backend/compiler/agents/high_param_submitter.py`) with Validator-owned acceptance/cleanup. - -### Overview - -After the body section is complete (before conclusion), the system enters a **Critique Phase** that collects validator-approved self-review notes. Accepted critiques are appended to the paper transparently instead of rewriting paper content. - -### Workflow - -1. **Critique Aggregation** (3 total attempts required): - - Rigor & Proofs Submitter generates peer review feedback on body section; legacy critique role fields are compatibility aliases - - **Decline Mechanism**: Submitter can assess "no critique needed" when body is academically acceptable (counts toward 3 total attempts) - - Validator validates critiques/declines (accept/reject with feedback loop) - - Target: 3 total attempts (accepted + rejected + declined attempts) - - Runs inside the compiler critique phase, not as a separate Aggregator workflow - -2. **Self-Review Append**: - - If at least 1 critique is accepted: append accepted critiques as `AI Self-Review and Limitations` - - If 0 critiques are accepted: move to conclusion without adding the section - - The section is placed after compiler/appended proof material when present, otherwise after conclusion - - Critiques never trigger partial rewrites, total rewrites, body clearing, title changes, or outline updates - -### Rationale - -**Why append rather than rewrite:** -- Preserves the validated paper content and proof placement. -- Keeps model-discovered limitations visible to readers. -- Avoids rewrite loops and accidental loss of correct content. -- Makes the AI self-review honest provenance rather than hidden revision pressure. - -### Decline Mechanism (Academically Acceptable Body) - -**Purpose**: Allow submitter to assess when body is academically acceptable and no substantive critique is needed. - -**Academically Acceptable Criteria**: -- No mathematical errors or unsound reasoning -- No missing proofs or incomplete arguments -- No logical gaps affecting correctness -- Structural organization is coherent -- All outline requirements are met -- Content aligns with paper title and goals -- Mathematical rigor meets academic standards - -**Behavior When Target Met**: -- If NO accepted critiques: transition directly to conclusion -- If accepted critiques exist: append `AI Self-Review and Limitations`, then transition to conclusion -- Rationale: With only 3 attempts, no early termination mechanism is needed - -### JSON Schemas - -**Critique Submission:** -```json -{ - "critique_needed": true, - "submission": "Specific critique, or empty string when critique_needed=false", - "reasoning": "Why critique is or is not needed" -} -``` - -**Critique Validation:** -```json -{ - "decision": "accept or reject", - "reasoning": "Detailed explanation", - "summary": "Brief summary if rejected (max 750 chars)" -} -``` - ---- - -## 9. COMPILER RIGOR PROMPTS (LEAN 4 THEOREM FLOW) - -**File:** `backend/compiler/prompts/rigor_prompts.py` - -**BODY-ONLY MODE**: The rigor loop runs only during body construction and is capped at 5 consecutive rigor cycles before yielding back to construction/review. Once body is complete (Conclusion exists in paper, or autonomous mode `autonomous_section_phase != "body"`), rigor mode is skipped. Gated by `_is_body_complete()` in the coordinator. - -**CONFIG GATE**: When `system_config.lean4_enabled = false`, every rigor cycle declines immediately (no Lean calls, no theorem proposals). The Lean 4 toolchain + Mathlib workspace is a hard prerequisite. - -### Four-Stage Architecture - -The rigor loop no longer edits paper text directly during discovery/formalization. Each rigor cycle runs four stages, with the coordinator owning inline validator attempts and appendix routing: - -**Stage 1: Theorem discovery (unvalidated)** — `build_rigor_theorem_discovery_prompt` -- Rigor & Proofs Submitter reads the full writing context (outline direct-injected, paper direct-injected when it fits, RAG for the rest per the offload priority excluding `compiler_outline.txt` + `compiler_paper.txt`). -- Sees `EXISTING VERIFIED PROOFS` block (from `proof_database.get_all_proofs()`) so it does not re-propose already-verified theorems. -- Sees `OPEN PROOF TARGETS` block (from `proof_database.get_recent_failure_hints()`) as optional retry candidates for the same high-impact target. -- Decides whether a user-prompt-relevant theorem is worth attempting. Decline ends the rigor cycle. -- Discovery is explicitly allowed to construct extension theorems from partial paper work, the current outline, supporting context, or the user prompt when helpful to paper construction and/or the user's goal. It is not limited to exact claims already present in the current paper. -- Discovery must classify `theorem_origin` as `existing_paper_claim`, `extension_from_partial_work`, or `extension_from_user_prompt`, and must set `placement_preference` to `inline` or `appendix_only`. Extension-derived theorems must use `appendix_only`. - -**Stage 2: Lean 4 formalization** — compiler rigor uses the serial `ProofFormalizationAgent.prove_candidate(max_attempts=5)` path; autonomous proof verification has its own parallel Phase-A pipeline with full-script plus tactic-script attempts -- Up to 5 Lean 4 attempts with error-feedback chaining (failing tactic + goal states + raw Lean diagnostics fed back into each retry). -- Emits proof progress events with `source_type="compiler_rigor"` so the existing autonomous-mode proof UI can display the flow. Keep frontend-consumed event names stable; `proof_verified` is reserved for the registered/stored proof event. -- All-5-fail: candidate is recorded via `proof_database.record_failed_candidate` (becomes a future open high-impact proof target) and the cycle ends as a decline. - -**Stage 3: Post-Lean integrity + novelty classification + persistence** — shared `validate_full_lean_proof_integrity` helper from `backend/shared/lean_proof_integrity.py`, then shared `assess_proof_novelty` helper from `backend/autonomous/core/proof_novelty.py` -- Rejects Lean-accepted proofs that introduce new fake proof devices (`axiom`, `constant`, `opaque`) not present in the source context. -- Statement mismatch is not a hard reject by itself; preserve the real Lean-accepted theorem under the actual proved statement and downshift/rank it during novelty/storage. -- Classifies the verified proof as novel or known. -- `register_verified_lean_proof()` stores it with `source_type="paper"`, `source_id=f"compiler_rigor:{session}"`, and duplicate detection before appending/broadcasting stored proof state. -- Novel proofs automatically enter the highest-priority direct-injection block on the next submitter instantiation (via `proof_database.inject_into_prompt`). -- Non-novel proofs stay in the database, visible through `/api/proofs/*` for future reference-selection UI flows. - -- **Stage 4: Placement routing (inline attempts OR appendix-only)** — `build_rigor_placement_prompt` -- If `placement_preference="appendix_only"`, inline placement is skipped and the verified theorem is appended directly to the Theorems Appendix with `placement_outcome="appendix_requested"`. -- If `placement_preference="inline"`, submitter proposes an inline edit that introduces the theorem with an explicit "verified in Lean 4, see Appendix A, " marker. -- Validator uses the new `rigor_lean_placement` mode: judges placement and narrative only; `rigor_check` is **forced to True** regardless of LLM output (Lean 4 is the source of mathematical truth). -- Up to 2 placement attempts; attempt 2 receives the validator's rejection feedback via `validator_rejection_feedback` field. -- On double rejection (or when attempt 1 is not produced), the theorem is appended to the **Theorems Appendix** via `paper_memory.append_to_theorems_appendix(...)` with `placement_outcome="appendix_fallback"`. Counts as a `rigor_acceptance` because the math is preserved. - -### Stage 1 JSON Schema (discovery) - -```json -{ - "needs_theorem_work": true, - "theorem_statement": "precise statement with explicit hypotheses", - "formal_sketch": "concrete Mathlib tactics / lemmas that look promising", - "source_excerpt": "2-6 sentences of motivating paper/outline/context/user-prompt basis", - "theorem_origin": "existing_paper_claim | extension_from_partial_work | extension_from_user_prompt", - "placement_preference": "inline | appendix_only", - "retry_existing_failure_id": "theorem_id from OPEN PROOF TARGETS if retrying, empty otherwise", - "reasoning": "why this theorem is the best target right now OR why no theorem" -} -``` - -Decline form: -```json -{ - "needs_theorem_work": false, - "theorem_statement": "", - "formal_sketch": "", - "source_excerpt": "", - "theorem_origin": "", - "placement_preference": "", - "retry_existing_failure_id": "", - "reasoning": "why declining" -} -``` - -Placement preference rule: `extension_from_partial_work` and `extension_from_user_prompt` MUST resolve to `appendix_only` even if the model emits `inline`. Existing paper claims may use `inline` when the theorem strengthens local prose, or `appendix_only` when it is useful but would distract from the body. - -### Stage 4 JSON Schema (placement) - -```json -{ - "proceed": true, - "operation": "replace | insert_after", - "old_string": "exact anchor text in current paper", - "new_string": "inline theorem intro with Lean 4 marker + appendix reference (NO Lean code inline)", - "reasoning": "why this placement" -} -``` - -### Placement Rules (enforced by the validator prompt) - -The validator rejects placement submissions that: -- Omit the "verified in Lean 4" marker or the appendix cross-reference. -- Paste the full Lean 4 source code into the paper body (the Lean proof lives in the appendix only). -- Place the theorem in a nonsensical location (outside the relevant section). -- Break surrounding narrative coherence. - -The validator MUST NOT re-evaluate: -- Mathematical correctness (Lean 4 already verified it). -- Proof soundness. -- Edge cases / hypotheses. - -### Theorems Appendix format - -Each entry written by `format_theorem_appendix_entry(...)` (helper in `backend/compiler/agents/high_param_submitter.py`): - -``` -Theorem (proof_XXX) [Novel | Known] - -Status: verified by Lean 4 () -Statement: -Lean 4 proof: - ---- -``` - -### Context layout - -- Outline: always direct-injected. -- Paper: direct-injected when it fits; otherwise RAG'd under `mode="rigor"` excluding `compiler_outline.txt` + `compiler_paper.txt`. -- RAG evidence: follows the offload priority (Shared Training DB → Local Submitter DB → Rejection Log → User Upload Files). -- EXISTING VERIFIED PROOFS block: compact `(proof_id, novel, statement)` tuples from `proof_database.get_all_proofs()`. -- OPEN PROOF TARGETS block: recent failure hints from `proof_database.get_recent_failure_hints(limit=5)` (`theorem_id`, statement, Lean error summary, and formalization blocker clues). These are retry context for the same high-impact target, not permission to pursue supporting lemmas. - -### Websocket events surfaced by the rigor flow - -Compiler rigor progress should be visible through the standard proof/compiler WebSocket stream with `source_type="compiler_rigor"` where applicable. Keep frontend-consumed event names stable, especially the registered-proof `proof_verified` event and standard compiler submission/acceptance/rejection/decline events, but avoid treating every intermediate proof progress notification as a permanent prompt-design invariant. - ---- - -## 10. WOLFRAM ALPHA TOOL (CONSTRUCTION MODE) - -**File:** `backend/compiler/agents/writer_submitter.py` - -Wolfram Alpha is exposed to the main writer as a real OpenAI-compatible tool only during `WritingSubmitter.submit_construction` (body / conclusion / introduction / abstract). It is NOT available in `outline_create`, `outline_update`, `review`, or the rigor loop. When `system_config.wolfram_alpha_enabled=false` (or the Wolfram client failed to initialize), the tool is not registered on the LLM call at all and construction collapses to the pre-Build-4 single-shot call. - -### Tool Schema - -```python -WOLFRAM_TOOL_SCHEMA = { - "type": "function", - "function": { - "name": "wolfram_alpha_query", - "description": "Query Wolfram Alpha to verify a mathematical or computational claim before writing it into the paper. ...", - "parameters": { - "type": "object", - "properties": { - "query": {"type": "string", "description": "Natural-language Wolfram Alpha query, e.g. 'Is pi algebraic?'"}, - "purpose": {"type": "string", "description": "Brief note on how the result will be used in the paper (audit trail)."} - }, - "required": ["query", "purpose"] - } - } -} -``` - -### Budget + Loop Semantics - -- **20 Wolfram calls per construction submission**, defined by `WOLFRAM_MAX_CALLS_PER_SUBMISSION` in `writer_submitter.py`. -- The submitter loop: call LLM with tools attached → execute each `tool_calls[]` entry via `wolfram_client.query(...)` → append a `role=tool` turn per call → re-call LLM. Repeat until (a) the LLM returns a non-tool message (final JSON construction submission) or (b) the 20-call budget is exhausted. -- On budget exhaustion, the coordinator appends a one-time user-role reminder ("You have used all 20 Wolfram Alpha calls for this submission. Finalize your JSON response now.") and re-calls the LLM with `tools=None` so the model must produce a final JSON response. -- Fallback: if the tool-loop raises, construction falls back to a plain single-shot `generate_completion` call so forward progress is never blocked by tool-loop failures. - -### Audit Trail - -The Wolfram call audit trail attached to `CompilerSubmission.metadata["wolfram_calls"]` is redacted by default and stores metadata such as lengths, hashes, and redaction flags instead of raw query/result text: -```json -[ - {"query_redacted": true, "purpose_redacted": true, "result_redacted": true, "query_hash": "...", "result_length": 1234}, - ... -] -``` - -The validator may see that Wolfram checks occurred, but does NOT receive raw Wolfram query/result text or re-query Wolfram. Logs/WebSocket events expose only redacted metadata. - -### Websocket Event - -Per Wolfram call, the submitter broadcasts: -```json -{ - "type": "compiler_wolfram_call", - "data": { - "task_id": "comp_writer_007", - "query_redacted": true, - "purpose_redacted": true, - "result_redacted": true, - "query_hash": "...", - "result_length": 1234, - "calls_used": 3, - "calls_remaining": 17, - "max_calls": 20 - } -} -``` - -The frontend's `CompilerLogs.jsx` renders redacted Wolfram metadata, not raw query or result previews. - -### Backend Tool-Call Plumbing - -`api_client_manager.generate_completion`, `OpenRouterClient.generate_completion`, and `LMStudioClient.generate_completion` all accept OpenAI-compatible `tools=[...]` and `tool_choice` kwargs. When a model ignores the tools entirely (returns no `tool_calls`), the submitter's loop cleanly terminates on the first iteration — identical to the single-shot path. - ---- - -## 11. LEAN PROOF SEARCH TOOL - -**File:** `backend/shared/proof_search/tool_adapter.py` - -`search_lean_proofs` is the shared OpenAI-compatible tool adapter for searching indexed MOTO proof history and SyntheticLib4 proof records. It is retrieval/navigation infrastructure only; it does not validate proofs, alter Lean gates, or replace MOTO proof registration. - -### Tool Schema - -```python -SEARCH_LEAN_PROOFS_TOOL_SCHEMA = { - "type": "function", - "function": { - "name": "search_lean_proofs", - "description": "Search MOTO local proof history and SyntheticLib4 proof records for prompt-relevant Lean proof patterns. ...", - "parameters": { - "type": "object", - "properties": { - "action": {"type": "string", "enum": ["overview", "search", "hydrate", "attest_usage"]}, - "query": {"type": "string"}, - "goal_statement": {"type": "string"}, - "lean_template": {"type": "string"}, - "imports": {"type": "array", "items": {"type": "string"}}, - "dependency_names": {"type": "array", "items": {"type": "string"}}, - "corpora": {"type": "array", "items": {"type": "string", "enum": ["moto", "manual", "autonomous", "leanoj", "syntheticlib4"]}}, - "verified_only": {"type": "boolean"}, - "include_partial": {"type": "boolean"}, - "include_failed": {"type": "boolean"}, - "novelty_filters": {"type": "array", "items": {"type": "string"}}, - "module_filters": {"type": "array", "items": {"type": "string"}}, - "source_filters": {"type": "array", "items": {"type": "string"}}, - "exclude_ids": {"type": "array", "items": {"type": "string"}}, - "limit": {"type": "integer", "minimum": 1, "maximum": 7}, - "cursor": {"type": "string"}, - "hydrate_lean_code": {"type": "boolean"}, - "search_mode": {"type": "string", "enum": ["auto", "exact", "lexical", "text", "semantic", "hybrid"]}, - "source": {"type": "string", "enum": ["moto", "manual", "autonomous", "leanoj", "syntheticlib4"]}, - "proof_id": {"type": "string"}, - "fingerprint": {"type": "string"}, - "session_id": {"type": "string"}, - "usage_attestation": { - "type": "object", - "properties": { - "retrieval_batch_id": {"type": "string"}, - "used_fingerprints": {"type": "array", "items": {"type": "string"}}, - "unused_fingerprints": {"type": "array", "items": {"type": "string"}}, - "used_proofs": {"type": "array", "items": {"type": "object", "properties": {"fingerprint": {"type": "string"}, "theorem_statement_hash": {"type": "string"}, "lean_code_hash": {"type": "string"}}}}, - "entire_code_used": {"type": "boolean"}, - "moto_artifact_hash": {"type": "string"}, - "usage_type": {"type": "string"} - } - } - }, - "required": ["action"] - } - } -} -``` - -### Semantics - -- `overview` returns the compact corpus map from the unified proof-search service. -- `search` returns at most 7 combined proof records and treats `autonomous` as the `moto` corpus alias. -- `hydrate` fetches one indexed proof record by source/proof ID or SyntheticLib4 fingerprint and may return full Lean code when available. -- `attest_usage` persists a local usage-attestation JSONL record for SyntheticLib4 whole-proof usage; when `entire_code_used=true`, each used proof must include `fingerprint`, `theorem_statement_hash`, and `lean_code_hash`. Online submission is a later integration step. -- Tool calls bypass Supercharge through the existing `tools`/`tool_choice` path. - ---- - -## EXACT STRING MATCHING SYSTEM - -**All compiler modes use exact string matching with automated pre-validation for document edits:** - -### Core Principles -- Submitters provide exact text (`old_string`) to identify edit locations -- The `old_string` is pre-validated to exist verbatim (exactly) in the document -- The `old_string` is pre-validated to be unique (appear only once) -- If exact match fails, system tries in order: Unicode normalization → whitespace normalization (multi-spaces) → all-whitespace normalization (collapses newlines/spaces/tabs to single space) → backslash normalization (collapses `\\\\cmd` → `\\cmd`, handles over-escaping) → consecutive fuzzy matching (conservative last resort) -- If match fails or is ambiguous after all attempts, pre-validation rejects immediately with clear feedback (before LLM validation) -- LLM validation then focuses on placement context and semantic appropriateness -- Industry-standard approach used by Cursor, Claude Code, and similar tools - -### JSON Schema for Edit Operations - -All compiler modes (construction, review, rigor, outline_update) use this schema: - -```json -{ - "operation": "replace | insert_after | delete | full_content", - "old_string": "Exact text to find (must exist verbatim and be unique)", - "new_string": "Replacement or insertion text", - "content": "Full content for display/logging", - "reasoning": "Explanation of the edit" -} -``` - -### Operation Types - -| Operation | old_string Required? | new_string | Use Case | -|-----------|---------------------|------------|----------| -| `replace` | YES (must be unique) | Required | Replace existing text with new text | -| `insert_after` | YES (must be unique) | Required | Insert new text immediately after old_string | -| `delete` | YES (must be unique) | Empty | Remove the old_string from document | -| `full_content` | NO (empty) | Required | Add entirely new section (no existing match needed) | - - -### Validation Rules -- **REJECT** if `old_string` is not found after trying: exact match → Unicode normalization → whitespace normalization → all-whitespace normalization → backslash normalization → consecutive fuzzy matching -- **REJECT** if `old_string` matches multiple locations (not unique) -- **REJECT** if `operation` doesn't match the intent (e.g., using "replace" for new content) -- Validator confirms content flows naturally at the edit location -- Validator ensures edits align with outline structure - -### Consecutive Fuzzy Matching Fallback (Conservative) - -**Triggers only when layers 1-4 (exact, Unicode normalization, whitespace normalization, backslash normalization) all fail.** - -**Purpose:** Handles remaining model escaping quirks not caught by prior layers while maintaining safety. - -**Requirements (ALL must be met):** -1. **85% Consecutive Characters**: At least 85% of `old_string` must appear consecutively in document -2. **Tail Anchor**: Last 5% of `old_string` (minimum 20 chars) must match EXACTLY -3. **Uniqueness**: Exactly ONE match meeting both criteria (rejects if ambiguous) -4. **Minimum Length**: `old_string` must be >= 20 characters - -**Safety Mechanisms:** -- Rejects if multiple matches meet the 85% + tail anchor criteria (ambiguous) -- Rejects if `old_string` < 20 characters (too short for reliable fuzzy matching) -- Logs warnings when fuzzy match is used (indicates model escaping quirk) -- Updates `old_string` to actual document text before validation proceeds - -**Example Scenarios:** -- Model over-escaped: `\\\\mathbb{Z}` → Document has `\\mathbb{Z}` → **FUZZY MATCH SUCCESS** -- Model under-escaped: `\mathbb{Z}` → Document has `\\mathbb{Z}` → **FUZZY MATCH SUCCESS** -- Ambiguous: `Let x = 5` appears 3 times → **REJECT** (not unique) - -**Implementation:** Dynamic programming algorithm finds longest consecutive substring, enforces tail anchor match. - -### Enhanced Diagnostic Logging (Developer Debug Output) -When exact string matching fails during pre-validation, the system outputs comprehensive diagnostics to logs: -- **Needle/Haystack previews**: First and last 200 characters of both search string and document -- **Issue detection**: Checks for whitespace differences, partial matches, line ending mismatches, leading/trailing whitespace -- **Explicit failure reporting**: When no match found (even normalized), outputs "NO_MATCH_FOUND" with common causes (model hallucination, outline vs paper confusion, outdated content reference) -- **Purpose**: Enables rapid debugging of why models reference non-existent text without requiring manual document inspection - -### Automatic Marker Integrity Check - -**Before every old_string validation**, the system automatically checks and repairs missing structural markers: - -- **When**: Before `_pre_validate_exact_string_match()` in validator's `validate_submission()` method -- **What**: Checks for all required markers (placeholders and anchors) in the document -- **How**: - - For paper operations: Calls `paper_memory.ensure_markers_intact()` to check paper placeholders and anchor - - For outline operations: Calls `outline_memory.ensure_anchor_intact()` to check outline anchor -- **Why**: Prevents old_string match failures caused by markers being pruned during normal operation -- **Result**: If markers were missing, they are added and the document is re-fetched before validation proceeds - -This automatic repair runs on EVERY validation attempt, ensuring markers remain intact throughout the paper construction process. This complements the resume-time placeholder check (`ensure_placeholders_exist()`). - ---- - -## JSON Escape Rules (Applied in All Prompt Files) - -All prompt files include standardized, numbered JSON escape rules for clarity: - -``` -JSON ESCAPING RULES FOR LaTeX: -LaTeX notation IS ALLOWED and EXPECTED - you must escape it properly in JSON: - -1. Every backslash in your content needs ONE escape in JSON - - To write \mathbb{Z} in content, write: "\\mathbb{Z}" in JSON - - To write \( and \), write: "\\(" and "\\)" in JSON - - Example: Write "\\tau" not "\tau", write "\\(" not "\(" - -2. Do NOT double-escape: \\\\mathbb is WRONG, \\mathbb is CORRECT - - WRONG: "\\\\mathbb{Z}" (double-escaped, won't match document) - - CORRECT: "\\mathbb{Z}" (single-escaped, matches document) - -3. For old_string in edits: copy text EXACTLY from the document, just escape backslashes - - If document has \mathbb{Z}, your JSON must have "\\mathbb{Z}" - -4. Quotes: Escape double quotes inside strings as \" - - Example: "He said \"hello\"" - -5. Newlines/Tabs: Use \n for newlines (not \\n), \t for tabs (not \\t) - - Example: "Line 1\nLine 2" creates two lines - -6. Valid JSON escapes ONLY: \", \\, \/, \b, \f, \n, \r, \t, \uXXXX (4 hex digits) -``` - -**Applied in:** -- `backend/compiler/prompts/outline_prompts.py` -- `backend/compiler/prompts/construction_prompts.py` -- `backend/compiler/prompts/rigor_prompts.py` -- `backend/compiler/prompts/review_prompts.py` -- `backend/aggregator/prompts/submitter_prompts.py` -- `backend/aggregator/prompts/validator_prompts.py` - -**Rationale**: Models often generate LaTeX notation (`\tau`, `\Delta`, `\[`, etc.) which creates invalid JSON escape sequences. The numbered format with examples makes it clearer for models to follow. The `sanitize_json_response()` function in `json_parser.py` handles remaining issues automatically. - -**CRITICAL - LaTeX IS Allowed**: The rules above apply to system prompts. Retry prompts (when JSON parsing fails) **DO NOT** tell models to avoid LaTeX - they guide proper escaping. This ensures mathematical content is preserved and exact string matching works correctly. See **Standard LaTeX-Focused Retry** section above for retry prompt design. - ---- - -## PART 3 - AUTONOMOUS RESEARCH MODE JSON SCHEMAS - -**File:** `backend/autonomous/prompts/` - -Part 3 introduces autonomous topic selection, brainstorm-to-paper workflows, and paper library management. All JSON schemas follow the same escape rules as Part 1 and Part 2. - ---- - -### 0. TOPIC EXPLORATION (Pre-Selection Candidate Brainstorm) - -**File:** `backend/autonomous/prompts/topic_exploration_prompts.py` - -**Purpose:** Before topic selection, collect 5 validated candidate brainstorm questions using the full Part 1 aggregator infrastructure (parallel submitters, batch validation up to 3). Uses `build_exploration_user_prompt()` to frame the standard aggregator as a candidate question generator. Candidate questions must first prefer avenues that aggressively attack the user's WHOLE question as stated, no partial solutions; a true impossible/no-valid-solution answer counts as directly answering the whole question; next-best-piece candidates are valid only when a whole-question attack is absolutely not possible in one superintelligence brainstorm. - -**Architecture:** Reuses `AggregatorCoordinator` — no custom JSON schemas. Standard aggregator submitter/validator prompts handle generation and validation. The exploration user prompt provides the framing context (research goal, existing brainstorms/papers, diversity requirement). - -**Standard Aggregator JSON Schemas Apply** (from Part 1 submitter/validator prompts). - ---- - -### 1. TOPIC SELECTION SUBMITTER - -**File:** `backend/autonomous/prompts/topic_prompts.py` - -**Function:** `get_topic_selection_json_schema()` - -``` -REQUIRED JSON FORMAT: -{ - "action": "new_topic | continue_existing | combine_topics", - "topic_id": "string - Required if action is continue_existing", - "topic_ids": ["array of topic_ids - Required if action is combine_topics"], - "topic_prompt": "string - Required if action is new_topic or combine_topics. The brainstorm question/avenue to explore", - "reasoning": "string - Why this is the best choice right now" -} - -FIELD REQUIREMENTS: -- action: MUST be one of: "new_topic", "continue_existing", "combine_topics" -- topic_id: Required ONLY if action is "continue_existing" -- topic_ids: Required ONLY if action is "combine_topics" (array of 2+ topic IDs) -- topic_prompt: Required if action is "new_topic" OR "combine_topics" -- reasoning: ALWAYS required - -EXAMPLES: - -New Topic: -{ - "action": "new_topic", - "topic_prompt": "Explore connections between modular forms and Galois representations in the context of the Langlands program", - "reasoning": "The existing brainstorms have covered L-functions and automorphic representations. Modular forms provide a concrete computational entry point to the Langlands correspondence that hasn't been explored yet." -} - -Continue Existing: -{ - "action": "continue_existing", - "topic_id": "topic_003", - "reasoning": "The brainstorm on reciprocity laws has only 7 submissions and has not yet covered explicit formulas or computational approaches. Continuing this topic will provide more complete understanding before moving to a new avenue." -} - -Combine Topics: -{ - "action": "combine_topics", - "topic_ids": ["topic_002", "topic_005"], - "topic_prompt": "Unified exploration of local and global class field theory with applications to the Langlands program", - "reasoning": "Topics 002 (local class field theory) and 005 (global reciprocity) are closely related and would benefit from unified treatment. Combining them will reveal deeper connections." -} -``` - ---- - -### 2. TOPIC VALIDATOR - -**File:** `backend/autonomous/prompts/topic_prompts.py` - -**Function:** `get_topic_validator_json_schema()` - -``` -REQUIRED JSON FORMAT: -{ - "decision": "accept | reject", - "reasoning": "string - Detailed explanation for the decision" -} - -FIELD REQUIREMENTS: -- decision: MUST be either "accept" or "reject" -- reasoning: ALWAYS required - detailed explanation - -EXAMPLE: -{ - "decision": "accept", - "reasoning": "The proposed topic on modular forms represents a valuable new avenue that complements existing brainstorms on L-functions. The submitter correctly identifies this as a concrete entry point to Langlands correspondence that hasn't been explored in prior topics." -} -``` - ---- - -### 3. BRAINSTORM COMPLETION REVIEW SUBMITTER - -**File:** `backend/autonomous/prompts/completion_prompts.py` - -**Function:** `get_completion_review_json_schema()` - -``` -REQUIRED JSON FORMAT: -{ - "decision": "continue_brainstorm | write_paper", - "reasoning": "string - Detailed explanation of assessment", - "suggested_additions": "string - If continue_brainstorm, what mathematical areas remain unexplored (optional)" -} - -FIELD REQUIREMENTS: -- decision: MUST be either "continue_brainstorm" or "write_paper" -- reasoning: ALWAYS required -- suggested_additions: Optional, but recommended if decision is "continue_brainstorm" - -EXAMPLES: - -Continue Brainstorm: -{ - "decision": "continue_brainstorm", - "reasoning": "While the brainstorm has covered fundamental aspects of modular forms and their Galois representations, there remain unexplored areas including explicit computational methods, connections to elliptic curves, and applications to specific cases of Langlands correspondence.", - "suggested_additions": "Explore explicit computations of Galois representations attached to modular forms, investigate connections to elliptic curves over number fields, examine specific cases of the Langlands correspondence for GL(2)" -} - -Write Paper: -{ - "decision": "write_paper", - "reasoning": "The brainstorm has thoroughly explored modular forms, Galois representations, L-functions, automorphic forms, and their interconnections in the context of Langlands program. The database contains 23 high-quality submissions covering theoretical foundations, computational aspects, and specific examples. Further submissions would likely be redundant. A comprehensive paper can now synthesize these insights." -} -``` - ---- - -### 4. COMPLETION SELF-VALIDATOR - -**File:** `backend/autonomous/prompts/completion_prompts.py` - -**Function:** `get_completion_self_validation_json_schema()` - -``` -REQUIRED JSON FORMAT: -{ - "validated": true | false, - "reasoning": "string - Why the assessment is or isn't accurate" -} - -FIELD REQUIREMENTS: -- validated: MUST be boolean (true or false) -- reasoning: ALWAYS required - -EXAMPLES: - -Validated True: -{ - "validated": true, - "reasoning": "The completion assessment accurately reflects the current state of the brainstorm. The database has indeed covered all major mathematical avenues for this topic, and further exploration would likely yield diminishing returns or redundant content. The decision to proceed to paper writing is justified." -} - -Validated False: -{ - "validated": false, - "reasoning": "Upon reflection, the completion assessment was premature. While the brainstorm has good breadth, there are still unexplored computational techniques and specific examples that would strengthen a resulting paper. The suggested additions are valuable and warrant continued brainstorming before paper compilation." -} -``` - ---- - -### 5. REFERENCE PAPER EXPANSION REQUEST - -**File:** `backend/autonomous/prompts/paper_reference_prompts.py` - -**Function:** `get_reference_expansion_json_schema()` - -``` -REQUIRED JSON FORMAT: -{ - "expand_papers": ["array of paper_ids to see full content"], - "proceed_without_references": false, - "reasoning": "string - Why these papers should be expanded OR why no papers meet the 'very useful' threshold" -} - -FIELD REQUIREMENTS: -- expand_papers: Array of paper IDs (can be empty) -- proceed_without_references: Boolean - set true if no papers are very useful -- reasoning: ALWAYS required - -EXAMPLES: - -Expand Papers: -{ - "expand_papers": ["paper_003", "paper_007", "paper_011"], - "proceed_without_references": false, - "reasoning": "Papers 003, 007, and 011 appear highly relevant based on their abstracts. Paper 003 covers class field theory which connects directly to our brainstorm on reciprocity laws. Papers 007 and 011 discuss Galois representations and modular forms respectively, both central to our upcoming paper. Need to see full content to assess their utility for reference." -} - -Proceed Without References: -{ - "expand_papers": [], - "proceed_without_references": true, - "reasoning": "After reviewing all existing paper abstracts, none meet the 'very useful' threshold for the upcoming paper on modular forms and Galois representations. The existing papers focus on different aspects of Langlands program (L-functions, automorphic forms) that don't provide direct reference value for this specific paper topic." -} -``` - ---- - -### 6. REFERENCE PAPER FINAL SELECTION - -**File:** `backend/autonomous/prompts/paper_reference_prompts.py` - -**Function:** `get_reference_selection_json_schema(max_papers)` - -``` -REQUIRED JSON FORMAT: -{ - "selected_papers": ["array of up to caller cap paper_ids"], - "reasoning": "string - Why these specific papers are very useful for the upcoming paper" -} - -FIELD REQUIREMENTS: -- selected_papers: Array of paper IDs (maximum = caller-provided cap, can be empty) -- reasoning: ALWAYS required - -CONSTRAINTS: -- Caller supplies the cap: 3 for the normal topic-cycle reference workflow, 6 for Tier 3 short-form reference selection -- Papers must be selected from those shown in expansion request - -EXAMPLE: -{ - "selected_papers": ["paper_003", "paper_007", "paper_011"], - "reasoning": "After reviewing full content, these three papers provide the most useful reference material: Paper 003 establishes the class field theory foundation needed for our reciprocity discussions. Paper 007's treatment of Galois representations will inform our theoretical sections. Paper 011's computational examples of modular forms will enhance our practical demonstrations. The other expanded papers, while relevant, overlap too much with our brainstorm content or cover tangential topics." -} -``` - ---- - -### 7. PAPER TITLE SELECTION - -**File:** `backend/autonomous/prompts/paper_title_prompts.py` - -**Function:** `get_paper_title_json_schema()` - -``` -REQUIRED JSON FORMAT: -{ - "paper_title": "string - The complete title for the mathematical research paper", - "reasoning": "string - Why this title appropriately captures the brainstorm content and differentiates from existing papers (if any)" -} - -FIELD REQUIREMENTS: -- paper_title: ALWAYS required - complete paper title -- reasoning: ALWAYS required - -EXAMPLE: -{ - "paper_title": "Modular Forms and Galois Representations in the Langlands Program: A Computational Perspective", - "reasoning": "This title accurately captures the core content of our brainstorm database, which extensively covers both modular forms and Galois representations with emphasis on computational approaches. The subtitle 'A Computational Perspective' differentiates it from the existing theoretical paper on Langlands correspondence in our library and reflects the practical examples and algorithms present in the brainstorm submissions." -} -``` - -**DISTINCTION FOR TITLE VALIDATION:** - -The validator MUST distinguish between these two concepts: - -| Concept | Definition | Validation Behavior | -|---------|------------|---------------------| -| **Brainstorm Submissions** | Raw research insights in "BRAINSTORM SUMMARY" - the SOURCE MATERIAL for the paper | Title SHOULD reflect this content - that's expected! DO NOT reject for similarity to brainstorm submissions | -| **Existing Papers from Brainstorm** | Previously completed Tier 2 papers listed in "EXISTING PAPERS FROM THIS BRAINSTORM" | ONLY reject if title is too similar to these existing papers | - -**Key Validation Rules:** -- If "EXISTING PAPERS FROM THIS BRAINSTORM: None" → There's nothing to differentiate from, accept if other criteria met -- A title being similar to brainstorm content is CORRECT behavior (it captures the source material) -- A title being similar to an existing paper is a rejection reason (would create redundancy) -- DO NOT reject simply because the title reflects brainstorm submission content - ---- - -### 8. PAPER REDUNDANCY REVIEW - -**File:** `backend/autonomous/prompts/paper_redundancy_prompts.py` - -**Function:** `get_paper_redundancy_json_schema()` - -``` -REQUIRED JSON FORMAT: -{ - "should_remove": true | false, - "paper_id": "string - The paper_id to remove (or null if should_remove is false)", - "reasoning": "string - Detailed explanation of why this paper should be removed OR why no removal is needed" -} - -FIELD REQUIREMENTS: -- should_remove: Boolean -- paper_id: Required if should_remove is true, null otherwise -- reasoning: ALWAYS required - -CONSTRAINTS: -- Maximum 1 paper can be removed per review cycle -- Conservative approach: when in doubt, do NOT remove - -EXAMPLES: - -Remove Paper: -{ - "should_remove": true, - "paper_id": "paper_005", - "reasoning": "Paper 005 on 'Basic Principles of Class Field Theory' is now redundant. Papers 003, 009, and 014 provide more comprehensive coverage of class field theory with deeper mathematical rigor and broader applications. Paper 005's unique contributions (elementary introduction) are minimal and the library would be stronger without this redundant entry. The other papers fully subsume its content." -} - -No Removal: -{ - "should_remove": false, - "paper_id": null, - "reasoning": "After reviewing all paper titles and abstracts, no papers are redundant. Each paper provides unique perspectives, covers distinct mathematical areas, or approaches common topics from different angles. The library maintains good diversity without unnecessary overlap." -} -``` - ---- - -### PART 3 PROMPT ASSEMBLY PATTERNS - -### 9. BRAINSTORM CONTINUATION DECISION - -**File:** `backend/autonomous/prompts/paper_continuation_prompts.py` - -**Function:** `get_continuation_decision_json_schema()` - -``` -REQUIRED JSON FORMAT: -{ - "decision": "write_another_paper | move_on", - "reasoning": "string - Detailed explanation of assessment" -} - -FIELD REQUIREMENTS: -- decision: MUST be either "write_another_paper" or "move_on" -- reasoning: ALWAYS required -``` - -**Context**: User prompt + brainstorm topic + brainstorm summary + prior papers from this brainstorm (title/abstract/outline) + paper count ("N of 3 maximum"). Does NOT include the full brainstorm DB or cross-topic reference papers. - -**Validation**: Topic validator validates with `build_continuation_validation_prompt()` via `override_prompt` parameter. - ---- - -### PART 3 PROMPT ASSEMBLY PATTERNS (continued) - -All Part 3 prompts follow similar assembly patterns to Part 1 and Part 2: - -```python -# Standard assembly order: -1. get_system_prompt() # Role and task description -2. "\n---\n" -3. get_json_schema() # JSON format requirements -4. "\n---\n" -5. f"USER RESEARCH GOAL:\n{user_prompt}" # ALWAYS direct injected -6. "\n---\n" -7. context # Context varies by role -8. if rag_evidence: "\n---\nRETRIEVED EVIDENCE:\n{rag_evidence}" -9. "\n---\n" -10. "Now generate your response as JSON:" -``` - -**Context Variations by Role:** - -**Topic Exploration (uses Part 1 Aggregator):** -- Aggregator user prompt = `build_exploration_user_prompt()` containing research goal, existing brainstorms, completed papers, diversity framing -- Standard aggregator submitter/validator prompts + context handling (shared training DB, rejection logs, RAG cycling) - -**Topic Selection Submitter:** -- **5 validated candidate brainstorm questions** from topic exploration (direct injection) -- All brainstorm topics with metadata -- All completed papers with title + abstract + word count -- Topic selection rejection history (last 5) - -**Topic Validator:** -- Same as topic selection submitter -- Proposed topic selection action - -**Brainstorm Submitters:** -- Uses Part 1 Aggregator context (brainstorm DB, rejection logs, etc.) -- **PLUS: Selected reference papers as user_files (enables compounding knowledge across research cycles)** - -**Brainstorm Validator:** -- Uses Part 1 Aggregator context -- **PLUS: Selected reference papers as user_files (for validation context)** - -**Completion Review:** -- Full brainstorm database -- Brainstorm metadata -- Completion feedback (last 5) - -**Reference Selection (Pre-Brainstorm - "initial" mode):** -- User's high-level research prompt -- Brainstorm topic prompt (topic not yet explored) -- Paper titles + abstracts (or full papers if expansion requested) -- Purpose: Select papers to inform brainstorm exploration (COMPOUNDING KNOWLEDGE) - -**Reference Selection (Pre-Paper Writing - "additional" mode):** -- User's high-level research prompt -- Completed brainstorm database (RAG) -- Already-selected papers (from pre-brainstorm selection) -- Remaining paper titles + abstracts (or full papers if expansion requested) -- Purpose: Add additional relevant papers based on brainstorm insights - -**Paper Title Selection:** -- Brainstorm summary (direct injection; no full brainstorm RAG) -- Existing paper titles and abstracts from same brainstorm -- Selected reference paper title/abstract summaries (if any, direct injection) -- Validated candidate titles from title exploration (direct injection) - -**Paper Compilation:** -- Uses Part 2 Compiler context (outline, paper, brainstorm DB, reference papers) - -**Paper Redundancy Review:** -- All paper titles + abstracts - ---- - -### JSON Escape Rules (Part 3) - -Part 3 follows the same JSON escape rules as Part 1 and Part 2 (see main section above for full details): - -``` -JSON ESCAPING RULES FOR LaTeX: -LaTeX notation IS ALLOWED and EXPECTED - you must escape it properly in JSON: - -1. Every backslash in your content needs ONE escape in JSON - - To write \mathbb{Z} in content, write: "\\mathbb{Z}" in JSON - - To write \( and \), write: "\\(" and "\\)" in JSON - -2. Do NOT double-escape: \\\\mathbb is WRONG, \\mathbb is CORRECT - -3. Quotes: Escape double quotes inside strings as \" - -4. Newlines/Tabs: Use \n for newlines (not \\n), \t for tabs (not \\t) - -5. Valid JSON escapes ONLY: \", \\, \/, \b, \f, \n, \r, \t, \uXXXX (4 hex digits) -``` - -**Applied in:** -- `backend/autonomous/prompts/topic_prompts.py` -- `backend/autonomous/prompts/completion_prompts.py` -- `backend/autonomous/prompts/paper_reference_prompts.py` -- `backend/autonomous/prompts/paper_title_prompts.py` -- `backend/autonomous/prompts/paper_redundancy_prompts.py` -- `backend/autonomous/prompts/proof_prompts.py` - ---- - -### 10. PROOF PROMPTS (Lean 4 Formal Verification) - -**File:** `backend/autonomous/prompts/proof_prompts.py` - -All proof prompts use `_json_only_footer(example)` which appends: -`"Respond with ONLY valid JSON. Do not use markdown fences. Escape backslashes correctly for JSON."` -All proof prompts pass `temperature=0.0`. - ---- - -#### 10a. PROOF FRAMING GATE - -**Function:** `build_proof_framing_gate_prompt(user_prompt)` - -**Purpose:** One-shot decision at autonomous start — decides whether the research program should activate the full proof pipeline. Errs on the side of `true` whenever formal proof can materially help the user's prompt. - -```json -{ - "is_proof_amenable": true, - "reasoning": "brief explanation" -} -``` - -**Field requirements:** -- `is_proof_amenable`: Boolean. `true` enables `PROOF_FRAMING_CONTEXT` injection into all subsequent submitter prompts and activates the `ProofVerificationStage` after each brainstorm/paper. `false` disables the proof pipeline for this session. -- `reasoning`: Always required. - ---- - -#### 10b. PROOF IDENTIFICATION (Theorem Discovery) - -**Function:** `build_proof_identification_prompt(user_prompt, source_type, source_id, source_content, source_title="")` - -**Purpose:** Impact-first user-prompt relevance gate that extracts only proof candidates expected to produce new/novel prompt-directed knowledge absent from standard references or Mathlib. Bounded source-title/brainstorm-topic metadata may steer relevance but must not be treated as instructions. This is not a known-knowledge-base builder. It rejects routine helpers, supporting lemmas, trivial/easy proofs, standard/textbook/Mathlib restatements, program-local firsts, minor reformulations/local formalizations, off-prompt curiosities, and single-tactic/routine proof goals. Candidates are ordered by direct impact on the user's prompt: direct solutions or impossibility results first, then decisive reductions, obstructions, and structural theorems that themselves make major progress on the requested problem. No artificial theorem-count cap. - -```json -{ - "has_provable_theorems": true, - "theorems": [ - { - "theorem_id": "thm_1", - "statement": "natural-language theorem statement", - "formal_sketch": "optional note about assumptions, notation, or likely Lean formalization strategy", - "expected_novelty_tier": "mathematical_discovery", - "prompt_relevance_rationale": "why proving this would directly solve, solve toward, or materially help solve the user prompt", - "novelty_rationale": "why this is absent from standard references or Mathlib and public/citable rather than a known-knowledge base entry", - "why_not_standard_known_result": "why this is not merely a textbook/Mathlib/routine helper result" - } - ] -} -``` - -**Field requirements:** -- `has_provable_theorems`: Boolean. `true` only when at least one prompt-relevant candidate is expected to be new or novel enough to justify Lean cost. -- `theorems`: Array of every prompt-relevant impactful candidate, ordered by direct impact on the user's prompt, with user-prompt solution attempts and user prompt + brainstorm topic solution attempts co-equal top priority when bounded brainstorm-topic metadata is present. Empty array when `has_provable_theorems` is `false`. -- `theorem_id`: Stable string identifier such as `"thm_1"`, `"thm_2"`, etc. -- `statement`: Natural-language theorem statement. Required. -- `formal_sketch`: Optional Lean formalization hints, assumptions, or notation notes. -- `expected_novelty_tier`: Required. One of `major_mathematical_discovery`, `mathematical_discovery`, `novel_variant`, or `novel_formulation`; `not_novel` candidates are skipped before Lean cost. -- `prompt_relevance_rationale`: Required. Explains how the proof directly solves, solves toward, or materially helps solve the USER RESEARCH PROMPT or USER PROMPT + BRAINSTORM TOPIC. -- `novelty_rationale`: Required. Explains why this is absent from standard references or Mathlib and public/citable rather than a background fact or program-local first. -- `why_not_standard_known_result`: Required. Explains why this is not merely a textbook, Mathlib, routine helper, or known-knowledge-base entry. - -**What to extract:** Direct solutions, impossibility results, decisive reductions, new obstructions, structural theorems, and other high-impact new/novel proof targets absent from standard references or Mathlib that materially help answer, support, or advance the USER RESEARCH PROMPT. Supporting lemmas are not extracted, even as fallback targets. - -**What to reject:** Off-prompt mathematical curiosities, routine helper lemmas, minor reformulations/local formalizations, local bookkeeping facts, algebra cleanup, coercion/monotonicity facts, standard Mathlib/textbook restatements, general verified background-library entries, results closable by routine proof search or a single tactic (`simp`, `omega`, `norm_num`, `decide`, `aesop`, `rfl`), tautologies, and definitional equalities. - ---- - -#### 10c. MATHLIB LEMMA SEARCH - -**Function:** `build_lemma_search_prompt(user_prompt, source_type, theorem_statement, formal_sketch, source_excerpt)` - -**Purpose:** Suggests existing Mathlib declaration names likely useful for proving the target theorem. Output is injected into the formalization prompt as `RELEVANT MATHLIB LEMMAS`. - -```json -{ - "lemma_names": [ - "Nat.add_comm", - "Nat.add_assoc" - ], - "reasoning": "brief explanation" -} -``` - -**Field requirements:** -- `lemma_names`: Array of 5–10 Mathlib declaration name strings when possible. Empty array if none are evident or theorem is too vague. -- `reasoning`: Always required. - ---- - -#### 10d. SMT TRANSLATION - -**Function:** `build_smt_translation_prompt(user_prompt, source_type, theorem_statement, formal_sketch, source_excerpt)` - -**Purpose:** Translates an arithmetic/SMT-amenable theorem into SMT-LIB v2 for Z3 pre-check. Only called when `smt_enabled` and the candidate passes the SMT-amenability heuristic. Result feeds tactic hints into Lean 4 formalization — never substitutes for Lean verification. - -```json -{ - "smtlib": "(set-logic QF_LIA)\n(declare-const n Int)\n(assert (not (= (+ n 0) n)))\n(check-sat)", - "reasoning": "Negate the target theorem so unsat means the theorem is valid." -} -``` - -**Field requirements:** -- `smtlib`: SMT-LIB v2 text encoding the **negation** of the theorem (so `unsat` = theorem is valid). Empty string `""` if a faithful SMT translation cannot be produced. -- `reasoning`: Always required. - -**Constraints:** Quantifier-free arithmetic fragments preferred. Do not invent assumptions not strongly implied by the theorem statement. - ---- - -#### 10e. PROOF FORMALIZATION (Full Script) - -**Function:** `build_proof_formalization_prompt(user_prompt, source_type, theorem_statement, formal_sketch, full_source_content, source_excerpt, prior_attempts, relevant_lemmas, smt_hint)` - -**Purpose:** Primary formalization path — generates complete Lean 4 source ready to compile. Up to 3 attempts per candidate, with the complete source brainstorm/paper as mandatory direct context, focused excerpt as a navigation aid only, and the full error-feedback chain from prior attempts injected on each retry. Preserves the theorem's non-trivial content; never weakens the statement just to compile. - -```json -{ - "theorem_name": "optional_lean_identifier", - "lean_code": "import Mathlib\n\n theorem ... := by ...", - "reasoning": "brief note about the formalization strategy" -} -``` - -**Field requirements:** -- `theorem_name`: Optional Lean identifier string (e.g. `"myTheorem"`). Can be empty. -- `lean_code`: Complete, runnable Lean 4 source including all needed imports. Must close every goal without `sorry` or `admit`. Required. -- `reasoning`: Brief note on formalization strategy. Always required. - -**Critical constraints:** -- `sorry` / `admit` anywhere → proof rejected, counts as a failed attempt. -- Axiomatizing the theorem's own concepts to make the goal trivial → rejected. -- Complete source brainstorm/paper is mandatory direct context; do not silently truncate it or replace it with the focused excerpt. -- If the full claim cannot be proved, do not replace it with a narrower/supporting/trivial lemma; submit the strongest faithful attempt at the selected high-impact target and let Lean feedback expose the blocker. -- PRESERVE the theorem's non-trivial content — do not simplify into a trivial identity to make it compile. - ---- - -#### 10f. PROOF FORMALIZATION (Tactic Script) - -**Function:** `build_proof_tactic_script_prompt(user_prompt, source_type, theorem_statement, formal_sketch, full_source_content, source_excerpt, prior_attempts, relevant_lemmas, smt_hint)` - -**Purpose:** Fallback formalization path after full-script attempts fail — returns a theorem header plus a decomposed tactic list. Up to 2 attempts. It receives the complete source brainstorm/paper as mandatory direct context, the focused excerpt as supplemental navigation context, and prior attempts from the full-script phase so the tactic path sees the full failure history. - -```json -{ - "theorem_name": "optional_lean_identifier", - "theorem_header": "theorem optional_lean_identifier (n : Nat) : n + 0 = n", - "tactics": [ - { - "tactic": "simpa using Nat.add_zero n", - "reasoning": "Close the goal with the standard right-identity lemma." - } - ], - "reasoning": "brief note about the tactic strategy" -} -``` - -**Field requirements:** -- `theorem_name`: Optional Lean identifier. Can be empty. -- `theorem_header`: Lean 4 theorem signature without proof body (no `:= by`). Required. -- `tactics`: Ordered array of tactic objects. Each entry must include `tactic` (Lean tactic string) and `reasoning` (short note). `sorry` / `admit` never allowed. -- `reasoning`: Overall strategy note. Always required. - -**Fallback behavior:** If the model returns a malformed tactic response (missing header or empty tactics), the agent falls back to one additional `_run_full_script_attempt` call. - ---- - -#### 10g. PROOF NOVELTY ASSESSMENT - -**Function:** `build_proof_novelty_prompt(user_prompt, theorem_statement, lean_code, existing_novel_proofs)` - -**Purpose:** Post-verification novelty gate — classifies a Lean-4-verified theorem into a novelty tier. Does NOT re-check validity. Errs on the side of recognizing novelty for results that required multi-step reasoning, non-trivial formalization work, or original proof strategy. - -```json -{ - "novelty_tier": "mathematical_discovery", - "reasoning": "brief explanation" -} -``` - -**Field requirements:** -- `novelty_tier`: One of `not_novel`, `novel_formulation`, `novel_variant`, `mathematical_discovery`, or `major_mathematical_discovery`. Any tier except `not_novel` enters the highest-priority direct-injection block for all subsequent brainstorm/paper submitters via `proof_database.get_novel_proofs_for_injection()`. `not_novel` proofs are stored in the database but not injected; manual "Try to Prove This" still preserves/appends non-duplicate Lean-verified `not_novel` proofs intentionally for user-visible RALPH-loop state and exact duplicate avoidance. -- `reasoning`: Always required. - -**Novelty tiers:** -- `novel_formulation`: The surrounding mathematical area may be known, but the exact theorem statement, formulation, or Lean 4 mechanization is absent from standard references or Mathlib and is independently publishable/citable. Program-local firsts do not qualify. -- `novel_variant`: A non-trivial reformulation, restructuring, generalization, different proof strategy, weaker hypotheses, stronger conclusion, or original composition based on known material. -- `mathematical_discovery`: A new theorem, bound, connection, structural insight, formally verified conjecture, or independently publishable/citable mathematical contribution. -- `major_mathematical_discovery`: A possible field-level breakthrough that may be competitive for a major prize or medal in a related field if confirmed and accepted by domain experts. This sits above ordinary `mathematical_discovery`. - -**Not novel:** Direct Mathlib restatement; trivial identity or tautology; closable by a single standard tactic (`simp`, `omega`, `norm_num`, `decide`, `rfl`); duplicates an already-stored novel proof. - ---- - -## System Requirements - -These core requirements apply across all prompt types: - -1. **Internal Content Warning**: All system prompts include the standardized skepticism warning block -2. **Concrete Format Examples**: Every prompt includes correct/wrong format examples with visual indicators -3. **Structured Rejection Feedback**: Validators use the standardized rejection format (Reason/Issue/What I Saw/Expected/Fix) -4. **Direct-Solution Preference**: Prompts should first prefer avenues that aggressively attack the user's WHOLE question as stated, no partial solutions. If the true answer is that the user's question is impossible or has no valid solution as stated, that counts as directly answering the whole question. If a whole-question attack is absolutely not possible in one superintelligence brainstorm, prompts may choose the next best necessary piece whose resolution would visibly advance the original question. Meta-phases such as topic exploration and paper title exploration still output candidates, but those candidates are judged by this whole-question-first policy instead of being rejected for not being solutions themselves. -5. **Compiler Outline Injection**: The compiler outline is always fully injected (never RAGed) for structural framework -6. **Temperature Policy**: Default `temperature=0.0`; only Supercharge candidates and parallel brainstorm submitter lanes may use explicit diversity temperatures. Validators, compiler roles, proof/final roles, and JSON retries stay `0.0`. -7. **JSON Preprocessing**: All LLM responses preprocessed by `sanitize_json_response()` -8. **Exact String Matching**: Document edits use exact verbatim matches with conservative consecutive fuzzy matching fallback for model escaping quirks (85% consecutive + tail anchor + uniqueness required) -9. **Phase-Based Construction**: Papers written in order: Body → post-body critique/self-review → Conclusion → Introduction → pre-abstract empirical red-team review → Abstract -10. **Required Sections**: - - **OUTLINE**: Must include Introduction, Body, Conclusion (Abstract is optional - can be "Abstract", "I. Abstract", or "0. Abstract") - - **PAPER CONSTRUCTION**: Final document order is Abstract → Introduction → Body → Conclusion plus optional appendix/self-review material, while writing order is Body → critique/self-review → Conclusion → Introduction → pre-abstract empirical red-team review → Abstract -11. **No Placeholder Output**: Submissions must never contain placeholder markers -12. **Placeholder Resume Repair**: When resuming from existing paper, missing placeholders are automatically added via `paper_memory.ensure_placeholders_exist()` to prevent "old_string not found" failures -13. **Fake Placeholder Detection**: System distinguishes real section content from model-inserted fake placeholder text (FULL content >300 chars = real; <300 chars with keywords = fake) to prevent confusion during marker repair - ---- +When the user explicitly directs a JSON or prompt-contract change, update the minimum necessary production prompt, parser/model, deterministic and semantic validation, and focused tests together. Update public API contracts and their version only when the externally consumed contract changes. diff --git a/.cursor/rules/latex-renderer.mdc b/.cursor/rules/latex-renderer.mdc index 26af79f..bf906d7 100644 --- a/.cursor/rules/latex-renderer.mdc +++ b/.cursor/rules/latex-renderer.mdc @@ -59,7 +59,7 @@ Both paths call `DOMPurify.sanitize(rawHtml, DOMPURIFY_CONFIG)` after `renderLat Dual rendering: **Rendered LaTeX View** (KaTeX math, dark theme on screen, white for PDFs) and **Raw Text View** (plain text, dark theme). All rendering flows through `LatexRenderer.jsx` (`frontend/src/components/LatexRenderer.jsx`). CSS in `LatexRenderer.css`. PDF generation via `downloadHelpers.js`. -**Performance Architecture**: For large documents (>~10K words), content is split into chunks at section boundaries. Each chunk renders independently via `IntersectionObserver`-gated `RenderedChunk` components — only chunks visible in the viewport (plus 600px margin) are LaTeX-rendered. Documents >50K chars auto-default to raw mode with a banner to switch. Content updates in rendered mode are debounced (1.5s) to prevent rapid re-rendering. +**Performance Architecture**: For large documents (>~10K words), content is split into chunks at section boundaries. Each chunk renders independently via `IntersectionObserver`-gated `RenderedChunk` components — only chunks visible in the viewport (plus 600px margin) are LaTeX-rendered. Paper views default to rendered mode, including documents >50K chars. Content updates in rendered mode are debounced (1.5s) to prevent rapid re-rendering. ### Props API @@ -97,7 +97,7 @@ DOMPurify sanitization is intentionally outside `renderLatexToHtml()` and must r **Debouncing**: `useDebouncedValue` hook delays rendered-mode processing by 1.5s during rapid content updates (WebSocket + polling). Raw mode is unaffected (instant updates). -**Auto-threshold**: Documents >50K chars (`LARGE_DOC_THRESHOLD`) auto-default to raw mode with a banner offering to switch to rendered view. +**Large-document threshold**: Documents >50K chars (`LARGE_DOC_THRESHOLD`) use the same progressive chunk rendering while preserving the caller's selected/default view. **Invariant**: Each chunk independently runs `renderLatexToHtml()` and then `DOMPurify.sanitize()`. Chunked and single-document paths must share the same conversion helpers and ordering anchors. @@ -123,9 +123,9 @@ DOMPurify sanitization is intentionally outside `renderLatexToHtml()` and must r | Component | Location | PDF | Toggle | Disclaimer | Notes | |-----------|----------|-----|--------|------------|-------| -| LivePaper.jsx | compiler/ | ✅ | ✅ | paper | Real-time paper viewing; auto-switches to raw >50K chars | +| LivePaper.jsx | compiler/ | ✅ | ✅ | paper | Real-time paper viewing; defaults to rendered | | PaperLibrary.jsx | autonomous/ | ✅ | ✅ | baked-in | Paper library cards (backend embeds disclaimer at save) | -| FinalAnswerView.jsx | autonomous/ | ✅ | ✅ | baked-in | Tier 3 final answer (defaults to raw for performance) | +| FinalAnswerView.jsx | autonomous/ | ✅ | ✅ | baked-in | Tier 3 final answer; defaults to rendered | | FinalAnswerLibrary.jsx | autonomous/ | ✅ | ✅ | paper | Final answer library (all sessions) | | LivePaperProgress.jsx | autonomous/ | ✅ | ✅ | paper | Live Tier 2 paper in progress | | LiveTier3Progress.jsx | autonomous/ | ✅ | ✅ | paper | Live Tier 3 paper in progress | diff --git a/.cursor/rules/main-rule-3-code-interaction-and-rule-interaction-rules.mdc b/.cursor/rules/main-rule-3-code-interaction-and-rule-interaction-rules.mdc index 45ec9f9..77d751c 100644 --- a/.cursor/rules/main-rule-3-code-interaction-and-rule-interaction-rules.mdc +++ b/.cursor/rules/main-rule-3-code-interaction-and-rule-interaction-rules.mdc @@ -17,17 +17,27 @@ alwaysApply: true 6.) For config/preset files with repeated literal values, never patch by replacing a shared literal alone. Anchor edits to the exact object/block being changed and verify the diff only touches the intended target. +6b.) Production prompt-engineering wording must not be edited, rewritten, shortened, generalized, reformatted, or “improved” without explicit user consent and direction for that prompt-language change. Permission to change adjacent code, schemas, models, validators, workflows, documentation, or rules is not implicit permission to alter prompts. When explicitly authorized, change only the requested minimum production wording and keep rule files limited to high-level, non-verbatim invariants. + 7.) Any REST shape, auth contract, `/api/features` capability, or web-consumed WebSocket contract change must update **code, the relevant rule(s), and `api_contract_version` in `/api/features`** in the same approved merge. The live backend's `GET /openapi.json` is the machine-readable REST schema contract. -7b.) SyntheticLib4 / unified proof-history search rule updates should stay high-level until the specific build phase lands. Exact JSON prompt schemas and prompt text are updated word-for-word only when the corresponding prompt/tool implementation is undertaken; planning docs alone do not require an API contract bump. +7b.) SyntheticLib4 / unified proof-history search rule updates should stay high-level until the specific build phase lands. Exact production prompt schemas or wording may change only under rule 6b; planning documents must not duplicate production prompt text and alone do not require an API contract bump. + +7c.) Desktop launcher frontend installs must follow the committed `frontend/package-lock.json`: use `npm ci` for fresh installs, non-destructive locked reconciliation for existing `node_modules`, read-only high/critical `npm audit` warnings, and never `npm audit fix` / `--force` on user machines. Dependency remediation belongs in the tested maintainer/CI release flow with an updated lockfile; if this invariant is broken, restore it before continuing launcher/updater work. + +7d.) Active/archive proof list, detail, and certificate REST records expose consistently normalized stable `run_id`, canonical `user_prompt`, independent novelty judgment, silent duplicate provenance, and versioned canonical artifact identity. Private-history duplication never enters the novelty prompt or suppresses current-run storage/context; prompt-novel external matches use the searchable `duplicate_novel` overlay and remain excluded from highest-priority current-run injection. SyntheticLib4 integrity hashes remain distinct from MOTO's canonical search hashes. + +7e.) Ordinary research, brainstorming, validation, and writing must pursue the user's exact objective with domain- and claim-appropriate rigor. Mathematics and formal proof remain first-class when relevant, but ordinary non-mathematical work must not be rejected merely for lacking mathematical form; explicit proof, Rigor & Proofs, and LeanOJ paths remain mathematical. -7c.) The desktop launcher `npm audit fix` remediation is permanent. Never remove, disable, weaken, or bypass the launcher code or rule that runs `npm audit fix` when `npm install` reports vulnerabilities; if it is accidentally removed or broken, restore it immediately with no exceptions before continuing launcher/updater work. +7f.) When Mathematical Proofs is allowed, the eager startup Autonomous LLM proof-framing gateway is essential: preserve its proof-amenability bias, persisted/resumed context injection, and the complete enabled Lean pipeline. When Mathematical Proofs is disabled in Allowed Outputs, skip the gateway and its proof-emphasis injection, including on resume. Never otherwise disable, bypass, defer, narrow, or weaken either without explicit user approval. -8.) Only ONE workflow mode may be active at a time (Aggregator, Compiler, Autonomous Research, or LeanOJ Proof Solver). This constraint applies identically in both default mode and generic mode. Start conflict checks must be serialized and include pending/background-task activity flags such as `autonomous_coordinator.is_active`, not only persisted `state.is_running` fields. +8.) Only ONE workflow mode may be active at a time (Aggregator, Compiler, Autonomous Research, or LeanOJ Proof Solver). This constraint applies identically in both default mode and generic mode. Start conflict checks must be serialized and include pending/background-task activity flags such as `autonomous_coordinator.is_active`, not only persisted `state.is_running` fields. A top-level Stop must serialize against that workflow's still-pending Start initialization before tearing down shared coordinator/RAG state. + +8a.) On Windows desktop/default mode, a committed top-level workflow owns a best-effort, non-blocking system-sleep inhibitor until explicit stop or terminal background cleanup. Generation-fenced ownership must survive same-run phase/task handoffs and reject stale releases. It prevents idle system sleep without forcing the display on; hosted/generic mode does not inhibit host power state. Logical workflow exclusion remains authoritative even if the native power API fails. 8b.) Autonomous Research and Single Paper Writer expose run-level Allowed Outputs (`allow_mathematical_proofs`, `allow_research_papers`); at least one must be true. Both true preserves existing workflow behavior. The Mathematical Proofs checkbox is the user-facing Lean proof-output enable path and must either sync/enable the runtime proof setting or the backend must reject proof-only/proof-requested starts when Lean is unavailable. Disabling papers must not disable brainstorming itself; proof-only autonomous runs must not silently become brainstorm-only loops and must reset durable workflow state to the next topic/exploration boundary after proof work instead of leaving `pre_paper_compilation`. Disabling proofs must skip proof-output work without affecting developer-only creativity boost behavior. -9.) Lean 4 and SMT features are gated by runtime flags: `lean4_enabled` gates Lean proof execution/model proof work, `lean4_lsp_enabled` only gates the optional persistent LSP optimization (subprocess Lean must still work when it is false), and `smt_enabled` gates Z3/SMT hint generation. All three default false; when disabled they must not invoke their corresponding toolchains, spend proof-model calls, or block workflows, and must never ship Lean or Z3 toolchains or Python wheels into `requirements-generic.txt`, `Dockerfile`, or `docker/entrypoint.sh` (hosted image stays Lean-free and Z3-free). Lean 4 is authoritative formal checking for every stored proof and is necessary for LeanOJ final solutions; SMT contributes hints only, and only valid `unsat` SMT checks become suggested Lean tactics. Z3 executable paths are trusted startup/operator configuration only, must be rejected as runtime API input, and must resolve to a `z3`/`z3.exe` executable. Automated proof candidates should directly serve the user prompt and prioritize high-impact prompt-solving targets before Lean cost, but once Lean accepts real proof code it must be preserved and novelty-ranked under the actual proved statement; statement mismatch downshifts storage instead of discarding. Do not spend proof attempts building a general known-knowledge base of supporting lemmas, routine helpers, standard Mathlib/textbook facts, minor reformulations/local formalizations, trivial/easy proofs, or merely non-trivial background lemmas. LeanOJ final master-proof edits may use standard facts inline to solve the template, but must not accumulate a separate known-knowledge library. +9.) Lean 4 and SMT features are gated by runtime flags: `lean4_enabled` gates Lean proof execution/model proof work, `lean4_lsp_enabled` only gates the optional persistent LSP optimization (subprocess Lean must still work when it is false), and `smt_enabled` gates Z3/SMT hint generation. All three default false; when disabled they must not invoke their corresponding toolchains, spend proof-model calls, or block workflows, and must never ship Lean or Z3 toolchains or Python wheels into `requirements-generic.txt`, `Dockerfile`, or `docker/entrypoint.sh` (hosted image stays Lean-free and Z3-free). Lean 4 is authoritative formal checking for every stored proof and is necessary for LeanOJ final solutions; SMT contributes hints only, and only valid `unsat` SMT checks become suggested Lean tactics. Z3 executable paths are trusted startup/operator configuration only, must be rejected as runtime API input, and must resolve to a `z3`/`z3.exe` executable. Automated proof candidates should directly serve the user prompt and prioritize high-impact prompt-solving targets before Lean cost, but once Lean accepts real proof code it must be preserved and novelty-ranked under the actual proved statement; statement mismatch downshifts storage instead of discarding. Lean formalization attempts may reuse the last available Assistant proof pack but must not refresh Assistant retrieval during the Lean attempt phase. Do not spend proof attempts building a general known-knowledge base of supporting lemmas, routine helpers, standard Mathlib/textbook facts, minor reformulations/local formalizations, trivial/easy proofs, or merely non-trivial background lemmas. LeanOJ final master-proof edits may use standard facts inline to solve the template, but must not accumulate a separate known-knowledge library. 10.) LeanOJ initial topic generation and brainstorm submitters always run in parallel and feed one validator that batch-validates up to 3 topics/submissions. Initial topic candidates/selection must be broad locked foundation questions covering the whole LeanOJ solution route, not narrow sublemma/tactic/local-repair topics. Recursive brainstorming has no separate recursive-topic prepass and must not re-inject the initial selected topic as active steering context; it uses the shared accepted proof-memory database plus the current proof/failure context. Accepted brainstorm memory must preserve occurrence-specific chronological metadata even for duplicate idea text. Never implement active LeanOJ topic or brainstorm phases as round-robin/serial submitter calls; one hung submitter must not halt the phase. 10b.) Developer-enabled LeanOJ Creativity Emphasis Boost applies to every fifth valid queued initial-topic and brainstorm submission per submitter. It only adds optional near-solution/adjacent-solution creativity pressure when apparent and potentially very helpful; validation remains unchanged, accepted/rejected WebSocket payloads mark `creativity_emphasized`, and the block is skipped for that slot if it would overflow the configured prompt budget. @@ -35,14 +45,20 @@ alwaysApply: true 11.) LeanOJ stop/crash/restart preserves resumable state by default. `Clear Progress` / `/api/leanoj/clear?confirm=true` is the only intentional reset path. Start/restart should choose the best matching/resumable persisted session by proof context, not blindly create a new session or pick the latest file; automatic model-work restart on backend boot is opt-in via runtime config. -12.) LeanOJ and autonomous proof-check recoverable provider-credit exhaustion should preserve workflow checkpoints and pause rather than become proof-attempt failures. Hard configuration/privacy/missing-key errors should fail visibly with a user-repair path instead of inflating proof attempt loops. `Retry OpenRouter` / `/api/openrouter/reset-exhaustion` wakes currently waiting in-process credit pauses after credits are restored; stopped/restarted runs resume through their persisted LeanOJ/proof checkpoint state. +12.) LeanOJ and autonomous proof-check recoverable provider-credit exhaustion should preserve workflow checkpoints and pause rather than become proof-attempt failures. Provider max-output truncation before usable Lean code is a failed proof attempt with live activity, not a research-stopping provider failure. Hard configuration/privacy/missing-key errors should fail visibly with a user-repair path instead of inflating proof attempt loops. `Retry OpenRouter` / `/api/openrouter/reset-exhaustion` wakes currently waiting in-process credit pauses after credits are restored; stopped/restarted runs resume through their persisted LeanOJ/proof checkpoint state. 13.) LeanOJ/RALPH final-proof loop checkpoints may be user-configurable feedback checkpoints or conservative no-progress/stale-edit watchdog handoffs; they must not mark success or discard the durable draft. LeanOJ start requests expose configurable phase caps (`max_initial_brainstorm_accepts`, `max_recursive_brainstorm_accepts`, `final_attempts_per_cycle`); `final_attempts_per_cycle` bounds failed final verification/edit attempts before the next path decision/handoff, so accepted `needs_more_time=true` edits can extend a cycle while they keep passing the intermediate Lean gate. The durable `master_proof.lean` is the authoritative working draft, and every accepted master-proof edit must pass an in-memory Lean gate before persistence: `needs_more_time=true` runs Lean with `sorry`/`admit` placeholders allowed but still requires parse/typecheck, template preservation, and no fake proof devices; `needs_more_time=false` runs Lean placeholder-free and then final semantic review against the user prompt/template before the run stops as verified. Final-proof mode is edit-only: it must not be offered, shown, or taught `stuck_needs_brainstorm`, raw `need_more_brainstorming`, failed-attempt counts, or any path transition. It may see the most recent 5 final attempts as compact execution feedback (Lean errors, stale edit rejections, JSON truncation, watchdog/no-progress notices) and optional metadata-only `search_lean_proofs` context so it can avoid repeating failed edits without turning the master proof into a known-knowledge library. Lean/template rejection, semantic-review rejection, conservative no-progress/stale-edit watchdog feedback, and validator rejection of non-progressive shortening edits must preserve the master proof and persist structured continuation feedback; non-user-forced no-progress handoffs should gather recursive brainstorm context before re-entering final mode. 14.) LeanOJ/RALPH final verification must remain placeholder-free, but Lean-accepted scaffolds containing `sorry`/`admit` and Lean-accepted non-final-ready code should be saved as partial/supporting proofs for future context. Partial/scaffold checks may use subprocess fallback even when LSP mode is enabled. Partial proofs are citeable incomplete references only; never count them as final verified solutions and never accept fake `axiom`/`constant`/`opaque` proof devices. LeanOJ proof-gated brainstorm `lean_proof` submissions are preserved once Lean and integrity checks pass, but brainstorm validation must still reject artifacts that do not directly discharge, split, or repair exact template obligations. -15.) Parent/user-selected phases have hierarchy precedence over child branches. When a parent phase starts (LeanOJ forced final loop, autonomous paper writing, Tier 3 final answer/final selection), lower-tier brainstorm/topic/path child tasks must stop or be ignored. LeanOJ `Skip Brainstorm` locks the run into the final loop until the configured final-attempt cycle is exhausted; model/path requests for more brainstorming cannot override that user action early. `Force Brainstorm` is a separate explicit user override that returns to recursive brainstorming while preserving proof progress. +15.) Parent/user-selected phases have hierarchy precedence over child branches. When a parent phase starts (LeanOJ forced final loop, autonomous paper writing, Tier 3 final answer/final selection), lower-tier brainstorm/topic/path child tasks must stop or be ignored. LeanOJ `Skip Brainstorm` and `Force Brainstorm` synchronously advance a private control generation before waiting; serialized control persistence and generation checks fence stale model, validation, prune, Lean, semantic-review, registration, and master-proof commits while preserving the durable draft. `Skip Brainstorm` locks the run into the final loop until the configured final-attempt cycle is exhausted; model/path requests for more brainstorming cannot override that user action early. `Force Brainstorm` is a separate explicit user override that returns to recursive brainstorming while preserving proof progress. 16.) LeanOJ prompt flows must guard formal/informal mismatches: treat the Lean template as the formal source of truth, do not silently reinterpret operations such as `Nat` subtraction, sanity-check exact-template formulas on small cases when feasible, and never claim Lean acceptance alone proves the informal problem unless that correspondence is justified. 17.) LeanOJ final answers must give an explicit formula/value when the template asks for one; supremum-style encodings of the feasible set are intermediate progress, not final readiness. + +18.) Current-run Mathematical Proofs tabs are flat per-proof views and must not group by or display the run's user prompt; proof statements, reasoning, and Lean code remain complete. Prompt/run collapsing and bounded prompt previews belong only to historical proof libraries. Manual live proofs and archived manual proof runs use separate tabs, and LeanOJ completed-proof history excludes the currently loaded run. + +19.) User-facing fatal `context_overflow_error` and nonfatal `proof_context_overflow` activity must identify configured and effective routes without exposing provider text/secrets. Events remain workflow-scoped, survive bounded instance-prefixed persistence, and avoid duplicate stops. Proof overflow defers that candidate for retry after configuration changes while sibling proof work continues. + +20.) Each top-level run may own exactly one Progressive Solution Path: one manual Aggregator→Compiler run, one Autonomous session, or one LeanOJ session. Activation uses a monotonic count of accepted brainstorm events—not retained post-cleanup memory—and a parent-controlled child never writes it. Before five acceptances the path is absent from prompts and UI. After activation, only existing semantic LLM validators may originate optional material updates in their normal response; malformed/omitted path JSON cannot affect the primary decision. Exact Main Submitter 1 review is serial, transactional, and non-blocking. Hard failures enter visible user-repair state and resume only through an explicit generation-fenced retry. Stop/crash preserves plan/queue; serialized Clear/new-run is the only reset. diff --git a/.cursor/rules/part-1-aggregator-tool-design-specifications.mdc b/.cursor/rules/part-1-aggregator-tool-design-specifications.mdc index 16657fe..1ebd2da 100644 --- a/.cursor/rules/part-1-aggregator-tool-design-specifications.mdc +++ b/.cursor/rules/part-1-aggregator-tool-design-specifications.mdc @@ -15,7 +15,7 @@ Configurable 1-10 submitters + exactly 1 validator (default 3 submitters). Each **Single Validator Constraint**: Only one validator allowed — multiple validators would cause divergent database evolution, breaking coherent Markov chain alignment. -Validator accepts a submission if adding it makes the training database more useful toward finding solutions. Submissions must first prefer avenues that aggressively attack the user's WHOLE question as stated, no partial solutions. If the true answer is that the user's question is impossible or has no valid solution as stated, that counts as directly answering the whole question. If a whole-question attack is absolutely not possible in one superintelligence brainstorm, they may choose the next best necessary piece whose resolution would visibly advance the original question. Broader exploratory/background-heavy avenues are allowed only when clearly required for that whole-question route, and easy/practical/broad/interesting detours must lose to a more direct rigorous route to the full prompt. On a fresh cleared run the accepted-submissions database starts blank; on normal desktop runs it reloads persisted accepted submissions, stats, and top-level manual run activity from the active data root. Manual read-only results/events views hydrate persisted data without requiring Start; autonomous/mini-aggregators must not write the manual event log. Distributor updates all submitters after each acceptance. +Validator accepts a submission if adding it makes the training database more useful toward finding solutions. Ordinary submissions aggressively pursue the strongest credible and genuinely novel solution to the user's exact objective, using the contribution form and claim-type rigor appropriate to the problem; mathematics, theorem discovery, proof, and formalization remain first-class whenever relevant, but non-mathematical work is not rejected merely for lacking mathematical form. Submissions must first prefer avenues that aggressively attack the user's WHOLE question as stated, no partial solutions. If the true answer is that the user's question is impossible or has no valid solution as stated, that counts as directly answering the whole question. If a whole-question attack is absolutely not possible in one superintelligence brainstorm, they may choose the next best necessary piece whose resolution would visibly advance the original question. Broader exploratory/background-heavy avenues are allowed only when clearly required for that whole-question route, and easy/practical/broad/interesting detours must lose to a more direct rigorous route to the full prompt. On a fresh cleared run the accepted-submissions database starts blank; on normal desktop runs it reloads persisted accepted submissions, stats, and top-level manual run activity from the active data root. Manual read-only results/events views hydrate persisted data without requiring Start; autonomous/mini-aggregators must not write the manual event log. Distributor updates all submitters after each acceptance. ## Queue Submissions and Overflow Behavior @@ -49,7 +49,7 @@ Validator processes 1, 2, or 3 submissions simultaneously using batch-specific p **Submission context injection**: Submitter context direct-injects first and offloads existing context to RAG when needed. Validator submissions under review are mandatory direct context; if a single or batch validation prompt is still too large after normal allocation, reject that validation batch with diagnostic feedback rather than indexing the pending submission as RAG. -**Upload/path enforcement**: Server-side validation of `.txt` only, 5 MB max, filename sanitization, path traversal rejection. Upload responses return logical filenames, not absolute host paths. Public Aggregator starts resolve `uploaded_files` only under `user_uploads`; internal autonomous reference-paper context may opt into trusted data-root file references via an explicit coordinator flag. +**Upload/path enforcement**: Server-side validation allows `.txt` and `.lean` text uploads only, 5 MB max, UTF-8/non-empty content, filename sanitization, path traversal rejection, and delete-by-logical-filename cleanup. Upload responses return logical filenames, not absolute host paths. Public Aggregator starts resolve `uploaded_files` only under `user_uploads`; internal autonomous reference-paper context may opt into trusted data-root file references via an explicit coordinator flag. ## Context Allocation @@ -65,7 +65,7 @@ All Aggregator offload order and source-exclusion rules are centralized in `rag- ## Role Selection -User selects model per role. Multiple roles can share a model. Models load with user-set context sizes. Aggregator settings also expose one shared `aggregator_assistant` memory-search LLM role for optional verified proof-memory support; built-in/default profiles set Assistant equal to the primary Validator, and Session History Memory disabled greys Assistant out. Assistant may provide up to 7 prior verified proofs, reuses useful packs for two eligible receiver reads before refresh, skips true no-external-history targets because it only performs proof-memory retrieval for now, uses run-scoped cooldown for repeated zero-useful or stagnant retrieval, shuts down only for repeated zero-useful retrieval in the current run, hides skip/backoff/shutdown turns while showing Assistant model-output failures in live activity, and never injects into validator prompts. +User selects model per role. Multiple roles can share a model. Models load with user-set context sizes. Aggregator settings also expose one shared `aggregator_assistant` memory-search LLM role for optional verified Lean proof-history support; it cannot redirect or mathematically reinterpret an unrelated parent goal, and returning no support is valid. Built-in/default profiles set Assistant equal to the primary Validator, and Session History Memory disabled greys Assistant out. Assistant may provide up to 7 prior verified proofs, reuses useful packs for two eligible receiver reads before refresh, skips true no-external-history targets because it only performs proof-memory retrieval for now, uses run-scoped cooldown for repeated zero-useful or stagnant retrieval, shuts down only for repeated zero-useful retrieval in the current run, hides skip/backoff/shutdown turns while showing Assistant model-output failures in live activity, and never injects into validator prompts. Per-role Supercharge is optional. When enabled for a submitter or validator, `api_client_manager.generate_completion()` runs 4 parallel full answer attempts for that role call, then a 5th same-model synthesis call and returns only the synthesis result. Supercharge candidate attempts intentionally use temperatures `[0.0, 0.2, 0.4, 0.8]` to diversify parallel outputs; synthesis remains `0.0`. Candidate attempts are sanitized to reusable visible answer text before synthesis; private thought/channel/control transcript text must never be fed back as feedback, brainstorm memory, or synthesis context. The synthesis prompt frames candidates as optional working material: the model may use one, combine several, ignore all, or write a stronger new answer, while preserving the original role output contract. If Boost applies to that role/task, all internal Supercharge calls use the Boost config first. Tool-call requests bypass Supercharge. @@ -93,6 +93,12 @@ Embeddings for RAG: Default mode uses LM Studio first, falls back to OpenRouter Accepted submissions database: never truncated. Live preview shows exact non-truncated log. Validator reasoning/results NOT included in the database (accepted submissions only). +## Progressive Solution Path + +Manual Aggregator owns one durable Progressive Solution Path for the whole manual Aggregator→Compiler run. It is absent from prompts and UI until the fifth accepted brainstorm event; acceptance counting is monotonic across Stop/Start, remains distinct from cleanup-reduced retained database size, and does not include validation, proof, or paper events. Normal Manual Aggregator owns this count; parent-controlled child Aggregators share the manager without count authority. After activation, the current Main Submitter 1-approved canonical revision proposed by the semantic validator is optional advisory direct context. + +Only the existing semantic validator may originate an optional update, as a separately parsed addition to its normal JSON, and only for material route/goal/evidence changes or meaningful step status—not routine nuance. Malformed or omitted path data cannot change the primary decision. Proposals are durably queued without blocking validation and are transactionally reviewed by a dedicated role copied exactly from Main Submitter 1. Only completely valid approved revisions become canonical; hard failures enter explicit user-repair state and resume only through the generation-fenced retry action. Stop/crash preserves the plan and pending work; explicit manual Clear removes it and is serialized against same-run reacquisition. + ## Database Cleanup Review Every 7th tracked acceptance (`total_acceptances % 7 == 0`, minimum 7 before first review). Manual Part 1 normally reloads persisted stats, so its cleanup cadence is persisted across restarts; autonomous/mini-aggregator runs can use fresh run-local stats when initialized with stats loading skipped. Autonomous resume offsets used for hard caps/completion reviews do not shift this cleanup modulo. @@ -115,7 +121,7 @@ New submission acceptance → synchronously persists to accepted memory and asyn ## Validation -JSON formatting used for all submission/validation communication. Submissions must be rooted in sound mathematical reasoning with no unfounded claims or logical fallacies. +JSON formatting is used for all submission/validation communication. Ordinary submissions must meet domain- and claim-appropriate standards of correctness and defensibility with no unfounded claims, fabricated support, or logical fallacies. Mathematical claims specifically require sound derivation, proof, or explicit assumptions. JSON validation failure: reject submission, send reason + content to submitter's local failure feedback memory. @@ -123,4 +129,4 @@ JSON validation failure: reject submission, send reason + content to submitter's When `lean4_enabled`, submitters may use `submission_type="lean_proof"` only for high-impact prompt-directed proof candidates whose claimed novelty is public/citable and absent from standard references or Mathlib, not merely new to the program. The shared Lean proof gate rejects missing/invalid novelty tiers and missing prompt-relevance/novelty/anti-standard-result rationales before Lean cost. Supporting lemmas, routine helpers, local facts, trivial/easy proofs, and weakened substitutes are not valid targets. Once Lean accepts real proof code, preserve/register the actual artifact for novelty/triviality ranking, but normal brainstorm validation may still reject it from the accepted-submission database when the actual theorem is low-impact, trivial, routine, or redundant. Hard rejection always applies to non-Lean-verified attempts, malformed submissions, or fake proof devices such as new `axiom`/`constant`/`opaque` declarations. -Manual Aggregator live results expose "Try to Prove This" over the current accepted-submissions database. It uses standard proof discovery, asking for candidates that directly solve the user prompt first and then candidates that substantially build toward solving it. When Session History Memory is enabled, the separate Assistant LLM role runs in parallel during brainstorming and proof checks, maintaining a freshness-tagged up-to-7 pack of fully Lean-verified local/SyntheticLib4 memory supports for eligible submitter/proof roles; proof discovery/formalization may use the latest pack but must not wait for it, and useful packs refresh only after two eligible receiver reads. Assistant cooldown state is run-scoped across transient task IDs/roles, survives ordinary stop/restart for that same run scope, is cleared only by explicit workflow/session reset, hides skip/backoff/shutdown turns from live activity, and shows Assistant model-output failures as errors; zero-useful retrieval can shut down for the run, while stagnant same-pack retrieval only backs off. Validators never receive Assistant context. Any non-duplicate Lean-verified proof from that user-triggered check is stored in the active manual proof database and appended to the active manual Aggregator database proof appendix for user-visible display/download, regardless of novelty rating. This known-proof preservation is intentional for RALPH looping: the user can see validated work, and later proof checks can avoid redoing exact verified proofs instead of losing them because they ranked `not_novel`. Progress is also persisted to the manual Aggregator event log so live activity can recover missed WebSocket events. Later proof-check prompts strip that generated-proof appendix and receive verified proofs through the manual proof database injection instead. Manual Aggregator clear archives the active manual proof database to history and resets active proof context to empty; clear is rejected while manual proof verification is active. +Manual Aggregator live results expose "Try to Prove This" over the current accepted-submissions database. It prioritizes direct prompt solutions, then substantial progress. Assistant retrieval is parallel/non-blocking, capped at 7, refreshed after two receiver reads, and filters the requesting run/session on fresh and reused packs; validators never receive it. Every newly verified current occurrence is stored after independent novelty judgment, while source appendices avoid only same-source exact duplication. Known and cross-run `duplicate_novel` proofs remain visible/searchable for RALPH history but stay outside highest-priority prompt-novel injection. Clear archives the active manual proof run and resets active proof context; it is rejected during verification. diff --git a/.cursor/rules/part-1-and-part-2-cointeraction-architecture.mdc b/.cursor/rules/part-1-and-part-2-cointeraction-architecture.mdc index f683312..db1a762 100644 --- a/.cursor/rules/part-1-and-part-2-cointeraction-architecture.mdc +++ b/.cursor/rules/part-1-and-part-2-cointeraction-architecture.mdc @@ -55,6 +55,7 @@ Parent workflow actions override child agents immediately. Manual paper writing, - Configurable 1-10 parallel submitters (default: 3) - Each submitter can have its own model, context window, and max output tokens - Enables multi-model exploration of different solution basins simultaneously +- Ordinary Aggregator submissions are domain-general and pursue the strongest credible, genuinely novel solution to the exact objective. Validators apply claim-type and domain-appropriate rigor; mathematics and formal proof remain first-class when relevant but are not a universal output form. - Parallel submitter generation uses the shared temperature ladder `[0.0, 0.1, ..., 0.9]` by submitter index; single-model sequential submitters and validators stay `0.0`. - Developer-enabled Creativity Emphasis Boost adds an optional creativity prompt block every fifth valid submission slot per Aggregator submitter, including Autonomous topic/title mini-aggregators and Tier 1 brainstorm aggregators; it does not alter validators or scheduling, and it is skipped for that slot if the extra block would overflow the configured context budget. - If all submitters and the validator are configured with the same model ID, the Aggregator normally uses single-model sequential mode regardless of provider. Exception: when all roles are LM Studio and LM Studio reports multiple loaded same-base numeric `:#` instances for that model, submitters may run in parallel and `lm_studio_client` routes independent calls to idle sibling instances while the validator remains ordered. @@ -71,8 +72,9 @@ Parent workflow actions override child agents immediately. Manual paper writing, ### Compiler Single-Submitter (Part 2) - Fixed sequential architecture (NOT multi-submitter configurable): - - **Writing Submitter**: Handles outline_create, outline_update, construction, review modes. During construction, may invoke the Wolfram Alpha tool up to 20 times per submission when `system_config.wolfram_alpha_enabled=true`. - - **Rigor & Proofs Submitter** (legacy `high_param_*` fields): Handles all proof-solving submitter work, compiler rigor mode, Lean formalization, theorem placement, and post-body critique/self-review generation. Rigor is the **Lean-4-verified-theorem flow**: impact-first user-prompt-relevant discovery (with expected novelty/prompt-relevance/anti-known-result rationale, and explicit extension theorems from partial paper work / outline / source brainstorm or aggregator context / user prompt when helpful) → up to 5 Lean 4 formalization attempts (with error feedback) → novelty classification → placement routing. Discovery and formalization see the current paper plus available source brainstorm/aggregator context and verified-proof summaries; they must not build a general known-knowledge base. Existing-paper-claim theorems may go through inline placement (2 attempts, validator uses `rigor_lean_placement` mode forcing `rigor_check=True`); extension-derived theorems are forced to `placement_preference="appendix_only"` and appended directly to the Theorems Appendix (`placement_outcome="appendix_requested"`). Inline failures still use Theorems Appendix fallback. The compiler writes verified proofs directly into the shared `proof_database` (same database used by autonomous mode); novel proofs automatically enter the highest-priority direct-injection block on the next submitter instantiation. + - **Writing Submitter**: Handles domain-general outline creation/update, construction, and review directed at the exact objective with domain- and claim-type-appropriate rigor. During construction, may invoke the Wolfram Alpha tool up to 20 times per submission when `system_config.wolfram_alpha_enabled=true`. + - **Rigor & Proofs Submitter** (legacy `high_param_*` fields): Handles all proof-solving submitter work, compiler rigor mode, Lean formalization, theorem placement, and post-body critique/self-review generation. Rigor is the **Lean-4-verified-theorem flow**: impact-first prompt-relevant discovery → up to 5 Lean attempts → independent novelty judgment without private history → silent cross-run canonical duplicate overlay → placement. Existing-paper claims may attempt inline placement; extension-derived theorems are appendix-only and inline failures fall back there. Every verified occurrence is stored; only independently prompt-novel non-duplicate tiers enter highest-priority injection, while `duplicate_novel` remains searchable. +- Ordinary post-body critique and semantic validation follow the same domain-general exact-objective standard. Mathematics, theorems, proofs, and LaTeX are first-class when relevant rather than mandatory for every paper; dedicated theorem discovery/formalization, Lean placement/checks, structural markers, and Wolfram behavior are unchanged. - Sequential Markov chain workflow (only one submission at a time) - Each compiler role has its own model, context, and max token settings (separate from aggregator) - UI shows Validator, Writing Submitter, Rigor & Proofs Submitter, and Assistant settings; standalone Critique Submitter settings are deprecated aliases of Rigor & Proofs. @@ -81,7 +83,9 @@ Parent workflow actions override child agents immediately. Manual paper writing, ## Assistant Proof-Retrieval Role -Aggregator, Compiler, Autonomous Research, and LeanOJ expose one shared `Assistant` LLM model role per workflow surface for non-blocking verified proof-memory retrieval. Assistant is not a submitter, validator, proof checker, workflow phase, or per-lane clone. Eligible main roles may consume the latest up-to-7 verified proof pack opportunistically, and useful packs are refreshed only after two eligible receiver reads; validators and dedicated critique phases never receive Assistant context. The configured Assistant LLM owns live pack selection, parent workflows never wait on it, and no-history targets are skipped because Assistant only performs proof-memory retrieval for now. Durable cooldown is run-scoped across transient task IDs/roles while preserving real source/session separation; zero-useful retrieval can shut down for the current run, while stagnant same-pack retrieval only backs off. Live activity should show normal Assistant retrieval result summaries and explicit Assistant model-output failures, not skip/backoff/shutdown turns. Retrieved supports are advisory: receivers must judge relevance, ignore unrelated supports, and never let Assistant packs broaden candidate eligibility, replace targets, or alter phase decisions. Built-in/default profiles copy Assistant settings from the primary Validator; if Session History Memory is disabled, Assistant is greyed out and does not run. +Aggregator, Compiler, Autonomous Research, and LeanOJ expose one shared `Assistant` LLM model role per workflow surface for non-blocking verified Lean proof-history retrieval. Assistant is not a submitter, validator, proof checker, workflow phase, or per-lane clone. Eligible main roles may consume the latest up-to-7 verified proof pack opportunistically, and useful packs are refreshed only after two eligible receiver reads; validators and dedicated critique phases never receive Assistant context. The configured Assistant LLM owns live pack selection, parent workflows never wait on it, and no-history targets are skipped because Assistant only performs proof-memory retrieval for now. Durable cooldown is run-scoped across transient task IDs/roles while preserving real source/session separation; zero-useful retrieval can shut down for the current run, while stagnant same-pack retrieval only backs off. Live activity should show normal Assistant retrieval result summaries and explicit Assistant model-output failures, not skip/backoff/shutdown turns. Retrieved supports are advisory: receivers must judge relevance, ignore unrelated supports, and never let Assistant packs redirect or mathematically reinterpret an unrelated goal, broaden candidate eligibility, replace targets, or alter phase decisions. Returning no proof support is valid. The WorkflowPanel Assistant Memory Bank may display the latest metadata-only support pack as novelty-colored clickable proof-history tiles. Built-in/default profiles copy Assistant settings from the primary Validator; if Session History Memory is disabled, Assistant is greyed out and does not run. + +Assistant candidate fusion excludes only occurrences carrying the persisted standalone exact-duplicate-emphasis purpose; ordinary `duplicate_novel` occurrences remain eligible. Cached/latest/stale/OAuth-cooldown packs are policy-versioned and re-filtered per occurrence, preserving a mixed fused support when any unmarked occurrence remains. The fused pool is capped at at most 64 before deterministic reduction to 21; the normal configured-provider path makes at most one non-Supercharged Assistant selection call for up to 7 supports, while unavailable/no-history/no-candidate/cooldown paths may make none. Live activity must not imply exactly 64 candidates were reviewed. ## Additional Traits Shared Between Aggregator-Submitters and Compiler-Submitters diff --git a/.cursor/rules/part-2-compiler-tool-design-specification.mdc b/.cursor/rules/part-2-compiler-tool-design-specification.mdc index 033e27a..40f289f 100644 --- a/.cursor/rules/part-2-compiler-tool-design-specification.mdc +++ b/.cursor/rules/part-2-compiler-tool-design-specification.mdc @@ -8,12 +8,20 @@ Main Architecture layout/design of the distillation/compiler portion of the two- ## Workflow Compiler Note Compiler runs independently from aggregator (manual start via API only). Strict Markov-chain: one active compiler role submits at a time, waits for validation/result handling before resuming. Only 1 submission in queue at a time. +## Progressive Solution Path + +Compiler continues the one durable manual-run Progressive Solution Path owned by Aggregator; compiler paper acceptances never activate it or create a second owner. Once the manual run has five accepted brainstorm submissions and a Main Submitter 1-approved revision, the bounded canonical plan is optional advisory direct context for relevant solving and semantic-validator prompts and may be ignored for a better route. Only existing Compiler semantic LLM validators may optionally propose material route/goal/evidence/meaningful-step updates in their normal JSON; this extension is parsed separately and cannot invalidate or change the paper decision. Exact-string checks, marker repair, Lean/tool/integrity results, submitters, and other deterministic checks never originate proposals. Dedicated exact-Main-Submitter-1 review is asynchronous, serial, and transactional; Stop/Start preserves state, while manual Clear deletes/resets the active path separately from proof-run archival. + +After backend restart, manual Compiler reconstructs that persisted manager from the durable Aggregator prompt/run identity and the latest available Main Submitter 1 configuration. A manual clear deletes only the active manual run's solution-path state. + ## Compile/Distillation Tool Outline Reads aggregator database + user prompt, distills into a single coherent paper. Runtime roles are Writing Submitter, Rigor & Proofs Submitter (legacy `high_param_*` fields), Validator, and optional Assistant. Main construction/rigor remains sequential; post-body critique generation is performed by Rigor & Proofs while critique validation/cleanup remain Validator-owned. Aggregator/brainstorm database material is high-priority optional source context, not a mandatory checklist. Compiler submitters may selectively use, synthesize beyond, or depart from database material when that better serves the user's prompt and remains rigorous. Validator must not reject solely for selective non-use of database material. +Ordinary outline, construction, review, post-body critique, and semantic-validation work is domain-general, attacks the user's exact objective, and applies rigor appropriate to the domain and claim type. Mathematics, theorem discovery, proofs, and LaTeX remain first-class when relevant, not universal admission requirements; dedicated theorem discovery/formalization, Lean placement/checks, marker handling, and Wolfram behavior remain unchanged. + **Context Anchors**: - **Paper Anchor**: `[HARD CODED END-OF-PAPER MARK -- ALL CONTENT SHOULD BE ABOVE THIS LINE]` - **Outline Anchor** (two lines): `[HARD CODED BRACKETED DESIGNATION THAT SHOWS END-OF-PAPER DESIGNATION MARK]` then `[HARD CODED END-OF-OUTLINE MARK -- ALL OUTLINE CONTENT SHOULD BE ABOVE THIS LINE]` @@ -32,9 +40,9 @@ Before every `_pre_validate_exact_string_match()`, system calls `paper_memory.en **Provider Selection**: Each visible compiler role (Validator, Writing, Rigor & Proofs, Assistant) can independently use LM Studio, OpenRouter, or desktop-only cloud providers such as OpenAI Codex OAuth, xAI Grok OAuth, or Sakana Fugu API key with optional LM Studio fallback for cloud providers (default mode). Deprecated `critique_submitter_*` request fields are compatibility aliases for Rigor & Proofs. OpenRouter keeps optional host-provider selection. In generic mode, all roles use OpenRouter only; LM Studio/OAuth options are hidden or unavailable. Built-in/default profiles set Assistant equal to the primary Validator unless the user edits it. -**Assistant memory support**: When Session History Memory is enabled, compiler/manual writer uses one shared `compiler_assistant` memory role, not one Assistant per writer/rigor/proof lane. Assistant runs beside eligible non-validator, non-critique compiler writing/review/rigor/proof roles as optional verified proof-memory support. It may provide up to 7 prior verified proofs, reuses useful packs for two eligible receiver reads before refresh, skips true no-external-history targets because it only performs proof-memory retrieval for now, and never blocks compiler progress. Durable cooldown is run-scoped across transient compiler task IDs/roles: zero-useful retrieval can eventually shut down for that run, while stagnant same-pack retrieval only backs off. Live activity should show normal Assistant retrieval result summaries and explicit Assistant model-output failures, not skip/backoff/shutdown turns. Compiler validators and dedicated critique/self-review phases never receive Assistant context. +**Assistant memory support**: When Session History Memory is enabled, compiler/manual writer uses one shared `compiler_assistant` role, not one per lane. It retrieves verified Lean proof history only; supports are optional and cannot redirect or mathematically reinterpret an unrelated writing goal. It provides up to 7 supports, never blocks progress, and refreshes after two receiver reads; every fresh or cached/latest/stale reuse excludes the requesting stable run/session and an emptied reused pack falls through to retrieval. Durable run-scoped cooldown behavior, failure visibility, and validator/critique exclusion remain unchanged. -**Allowed Outputs**: Single Paper Writer start requests include `allow_mathematical_proofs` and `allow_research_papers`; at least one must be true. Both true preserves today's paper-writing plus optional proof behavior. The Mathematical Proofs checkbox is the user-facing proof-output enable path and must not imply proof work when Lean is unavailable; proofs-only starts should reject clearly if Lean is disabled/unavailable. Papers-only suppresses rigor/save-time proof work for that run. Proofs-only runs proof verification over the current Aggregator database instead of compiling a paper, exposes running/stoppable status while the background proof check is active, and remains separate from developer-mode Creativity Emphasis Boost. Manual writer proof output (rigor proofs, save-time checks, proof-only checks, and "Try to Prove This") stores in the active manual proof database and appears in the Manual Mathematical Proofs tab, not in the autonomous session proof tab/activity/graph. Manual Aggregator/Compiler clear actions archive the active manual proof database into manual proof-run history, reset active proof context to empty, and remove the manual Aggregator proof appendix when needed, so old proofs never seed a new manual run; clears are rejected while manual proof verification is active. Manual proof checks always use standard proof discovery: candidates that directly solve the user prompt first, then candidates that substantially build toward solving it. When Session History Memory is enabled, the Assistant role may run in parallel for manual proof checks and live-paper proof checks, but checks continue without waiting for Assistant. Manual Aggregator proof checks (button or proofs-only Single Paper Writer) append any non-duplicate Lean-verified proof to the active Aggregator proof appendix for live display/download and future proof checks in the same active run, regardless of novelty rating; preserving known verified proofs here is intentional RALPH-loop state and exact duplicate avoidance, not a bug. Proof-check source prompts strip existing generated-proof appendices because verified proofs enter through the active manual proof database injection. Manual Aggregator proof checks must recover the persisted manual Aggregator prompt when the live coordinator prompt is unavailable after stop/restart. Manual `Try to Prove This` role resolution must use the active/manual Aggregator or Compiler role settings, including validator and Assistant settings, and must not fall back to autonomous proof runtime snapshots. The manual live paper exposes a read-only "Try to Prove This" check over the current paper plus current Aggregator context; it appends any non-duplicate Lean-verified proof to the live paper proof appendix regardless of novelty rating for the same manual RALPH-loop reason, while save-time proof checks append novel proofs to the saved paper text file. +**Allowed Outputs**: Single Paper Writer start requests include `allow_mathematical_proofs` and `allow_research_papers`; at least one must be true. Both true preserves today's paper-writing plus optional proof behavior. The Mathematical Proofs checkbox is the user-facing proof-output enable path and must not imply proof work when Lean is unavailable; proofs-only starts should reject clearly if Lean is disabled/unavailable. Papers-only suppresses rigor/save-time proof work for that run. Proofs-only runs proof verification over the current Aggregator database instead of compiling a paper, exposes running/stoppable status while the background proof check is active, and remains separate from developer-mode Creativity Emphasis Boost. Manual writer proof output (rigor proofs, save-time checks, proof-only checks, and "Try to Prove This") stores in the active manual proof database and appears as a flat per-proof list in the Manual Mathematical Proofs tab, not in the autonomous session proof tab/activity/graph. Archived manual runs are grouped by run/prompt in the separate Completed Proof Works history tab; only historical prompt headers are bounded previews, while proof content remains complete. Manual Aggregator/Compiler clear actions archive the active manual proof database into manual proof-run history, reset active proof context to empty, and remove the manual Aggregator proof appendix when needed, so old proofs never seed a new manual run; clears are rejected while manual proof verification is active. Manual proof checks always use standard proof discovery: candidates that directly solve the user prompt first, then candidates that substantially build toward solving it. When Session History Memory is enabled, the Assistant role may run in parallel for manual proof checks and live-paper proof checks, but checks continue without waiting for Assistant. Manual Aggregator proof checks (button or proofs-only Single Paper Writer) append any non-duplicate Lean-verified proof to the active Aggregator proof appendix for live display/download and future proof checks in the same active run, regardless of novelty rating; preserving known verified proofs here is intentional RALPH-loop state and exact duplicate avoidance, not a bug. Proof-check source prompts strip existing generated-proof appendices because verified proofs enter through the active manual proof database injection. Manual Aggregator proof checks must recover the persisted manual Aggregator prompt when the live coordinator prompt is unavailable after stop/restart. Manual `Try to Prove This` role resolution must use the active/manual Aggregator or Compiler role settings, including validator and Assistant settings, and must not fall back to autonomous proof runtime snapshots. The manual live paper exposes a read-only "Try to Prove This" check over the current paper plus current Aggregator context; it appends any non-duplicate Lean-verified proof to the live paper proof appendix regardless of novelty rating for the same manual RALPH-loop reason, while save-time proof checks append novel proofs to the saved paper text file. **Supercharge**: Each compiler role has a developer-mode-only Supercharge checkbox. Checked roles run 4 full answer attempts plus a 5th same-model synthesis answer through `api_client_manager.generate_completion()`. If Boost applies, every internal Supercharge call uses the Boost route/model/provider settings first. Tool-call requests bypass Supercharge; this is especially important for the Wolfram-enabled construction loop. @@ -112,7 +120,7 @@ The rigor loop no longer rewrites prose. Each rigor cycle: - Stage 1 output includes `theorem_origin` (`existing_paper_claim`, `extension_from_partial_work`, `extension_from_user_prompt`), `placement_preference` (`inline`, `appendix_only`), `expected_novelty_tier`, `prompt_relevance_rationale`, `novelty_rationale`, and `why_not_standard_known_result`. Invalid/missing novelty tiers or rationales decline before Lean cost. Extension-derived theorems MUST be forced to `appendix_only`; existing-paper-claim theorems may be inline or appendix-only. - Stage 2: `ProofFormalizationAgent.prove_candidate(max_attempts=5)` - up to 5 Lean 4 attempts with error-feedback chaining and complete current-paper source plus available source brainstorm/aggregator context as mandatory paper-writing proof context; focused excerpts are supplemental only. Routine workflow-memory retrieval is owned by the parallel Assistant LLM role when Session History Memory is enabled: Assistant observes the whole user prompt plus current phase/candidate/draft, rejection feedback, and Lean errors when relevant, then maintains a freshness-tagged up-to-7 pack of fully Lean-verified memory supports from local MOTO/manual/LeanOJ history and authorized SyntheticLib4 sources. Formalization receives the latest pack opportunistically and never waits for Assistant; useful packs refresh only after two eligible receiver reads. Direct `search_lean_proofs` prefetch/tool calls are explicit legacy/debug or narrow emergency-repair paths only. If full SyntheticLib4 Lean code is shown to a successful formalization prompt, MOTO records a local `model_visible_context` usage attestation without claiming whole-code dependency use. On 5 failures: record the candidate via `proof_database.record_failed_candidate` so future cycles see it as an open high-impact proof target; end the rigor cycle as a decline. - Proof-search retrieval is supporting context only: it must not turn rigor into a known-knowledge-base builder, alter the serial rigor loop, or bypass MOTO proof registration and integrity rules. -- Stage 3: hard post-Lean integrity checks reject only fake proof devices such as new `axiom`/`constant`/`opaque`; statement mismatch is non-blocking and downshifts storage to the actual Lean-verified theorem. Novelty classification and persistence go through the shared `register_verified_lean_proof()` path, which ranks preserved proofs and stores novel/non-novel records with duplicate detection under the active paper source id. Rigor discovery sees compact verified-proof summaries from the active proof database (autonomous session or active manual run), while source reads strip appended generated-proof sections so proof code does not duplicate through the paper/brainstorm source context. Canonical proof records and user-visible appendices remain preserved. Non-novel proofs remain available through `/api/proofs` for future user-driven reference selection. +- Stage 3: hard post-Lean integrity checks reject only fake proof devices such as new `axiom`/`constant`/`opaque`; statement mismatch downshifts storage to the actual Lean-verified theorem. Shared registration first performs novelty judgment without private proof history and persists every current-run occurrence, then silently overlays `duplicate_novel` only for independently prompt-novel canonical theorem+Lean matches from another run; the independent tier/reasoning remain provenance and `not_novel` is unchanged. Rigor discovery sees compact active proof summaries, generated-proof appendices are stripped from source context, and duplicate-novel occurrences remain searchable but outside highest-priority current-run injection. - Stage 4: placement - if `placement_preference="inline"`, HP model proposes an inline edit that introduces the theorem with an explicit "verified in Lean 4" marker and an appendix cross-reference. Validator uses `rigor_lean_placement` mode which forces `rigor_check=True` (Lean 4 is the source of mathematical truth) and judges placement/narrative only. Up to 2 placement attempts (attempt 2 gets validator rejection feedback). - Appendix routing: if `placement_preference="appendix_only"`, skip inline placement and append directly to the **Theorems Appendix** with `placement_outcome="appendix_requested"`. If inline placement is attempted but both placement attempts fail, append with `placement_outcome="appendix_fallback"`. Both outcomes count as `rigor_acceptance` because the math is preserved. - Loop 2 ends on first **decline** (no theorem found OR 5 Lean attempts failed OR Lean 4 disabled) or after 5 consecutive successful rigor cycles. Every verified theorem lands somewhere so there is no "rejection" outcome at the loop level. diff --git a/.cursor/rules/part-3-autonomous-research-mode.mdc b/.cursor/rules/part-3-autonomous-research-mode.mdc index acc147a..e7c822e 100644 --- a/.cursor/rules/part-3-autonomous-research-mode.mdc +++ b/.cursor/rules/part-3-autonomous-research-mode.mdc @@ -7,7 +7,7 @@ alwaysApply: true ## Overview -The Autonomous Research Mode is Part 3 of the MOTO Math Variant system. It is a self-directing three-tier research system that autonomously generates brainstorm topics, builds knowledge databases, produces complete mathematical research papers, and can synthesize a final answer based on a high-level research topic centered around the user prompt. +The Autonomous Research Mode is Part 3 of MOTO's self-directing three-tier research system. It autonomously generates brainstorm topics, builds knowledge databases, produces rigorous solution-oriented research papers or reports, and can synthesize a final answer centered on the user's high-level objective. Mathematics and formal proof remain first-class when relevant. **Example User Prompt**: "Solve the Langlands Bridge problem" or "Advance understanding of the Riemann Hypothesis" @@ -17,21 +17,33 @@ The Autonomous Research Mode is Part 3 of the MOTO Math Variant system. It is a - Part 3 (Autonomous Research) self-directs topic selection, brainstorming, and paper generation **Three-Tier Architecture**: -- **Tier 1**: Brainstorm aggregation databases (mathematical concept exploration) -- **Tier 2**: Finished mathematical research papers (compiled from brainstorm databases) +- **Tier 1**: Brainstorm aggregation databases (solution concept, mechanism, evidence, algorithm, and mathematical exploration) +- **Tier 2**: Finished rigorous research papers or solution reports (compiled from brainstorm databases) - **Tier 3**: Final answer synthesis (short-form answer or long-form volume from Tier 2 papers) ## Design Philosophy **Self-Directing Research**: The AI autonomously identifies the most valuable research avenues based on the high-level goal prompt. -**Basin Exploration**: Each brainstorm topic represents a "basin" of related mathematical concepts. The system explores each basin until sufficiently complete, then generates a paper. +**Domain-General Strategy**: Ordinary topic exploration/selection, completion review, reference selection, title selection, and continuation aggressively pursue the strongest solution path for the exact objective using domain- and claim-appropriate rigor. When Mathematical Proofs is allowed, mathematics and formal proof remain first-class when useful under the eager startup proof-framing emphasis; when disabled in Allowed Outputs, that framing is skipped. Positive proof framing is emphasis, not a hard mathematical admission requirement for every ordinary contribution. + +**Basin Exploration**: Each brainstorm topic represents a "basin" of related solution concepts, mechanisms, evidence, algorithms, designs, or mathematical structures. The system explores each basin until sufficiently complete, then generates a paper. **Cumulative Knowledge**: All brainstorm databases and papers persist, building a comprehensive research library over time. **Model Weight Exploration**: Completion review uses SPECIAL SELF-VALIDATION MODE because only the same model can assess whether its own weights have been exhausted for a given topic. -**External Verification Allowed**: The autonomous system may use the model's pre-trained mathematical knowledge, RAG context from prior work, user prompt, and external verification/search when the selected model/provider supports it. Internal AI-generated context remains non-authoritative and should be treated skeptically. +**External Verification Allowed**: The autonomous system may use the model's domain and mathematical knowledge, RAG context from prior work, user prompt, and external verification/search when the selected model/provider supports it. Internal AI-generated context remains non-authoritative and should be treated skeptically. + +## Progressive Solution Path + +Each autonomous research session owns exactly one durable Progressive Solution Path across all topics, brainstorms, papers, proof checkpoints, and Tier 3 phases. It remains absent from prompts and UI until five cumulative accepted Tier 1 brainstorm submissions in that session; topic/title mini-aggregator acceptances, paper edits, validator decisions, and proof events do not count or create independent owners. After activation, only the bounded Main Submitter 1-approved canonical revision is optional advisory direct context and may be ignored for a better prompt-serving route. + +The activation count is a persisted, monotonic session-wide counter of real Tier 1 accept events, distinct from the post-cleanup retained submission count. The autonomous parent is its sole writer; child Aggregators receive the shared manager for context/proposals without writing topic-local counts. It never resets at topic boundaries and resumes from durable metadata/workflow/solution-path state. + +Existing semantic LLM validators may separately propose a material route, goal, evidence, or meaningful-step-status update in their normal JSON response. Missing/malformed extensions do not affect the primary result; selectors, submitters, deterministic checks, Lean/SMT/tools, and integrity gates do not originate proposals. Proposals queue durably and non-blockingly for serial transactional review under the exact Main Submitter 1 provider/model/host/reasoning/fallback/context/output/Supercharge configuration. Hard failures enter user-repair state and resume only through the explicit generation-fenced retry action. Stop/crash/session resume preserves plan and queue; explicit session Clear/new-run removes them and is serialized against reacquisition. + +The Workflow panel hides the path before activation. After activation it distinguishes no-approved-plan, queued/reviewing updates, approved plans, and repair-required proposals; the typed snapshot separately exposes active/resumable ownership. Snapshot/event updates are fenced by run identity and lifecycle generation so stale cross-run responses cannot replace the current view. --- @@ -145,7 +157,6 @@ Topic exploration runs before EVERY new topic selection cycle — no exceptions. The autonomous topic submitter decides what to work on next. It can: 1. **Generate a NEW topic**: Identify the next most valuable brainstorm avenue 2. **Continue an EXISTING topic**: Resume work on an incomplete brainstorm -3. **Combine EXISTING topics**: Merge multiple related brainstorms into a unified topic ### Topic Submitter Context The submitter receives: @@ -166,15 +177,14 @@ The submitter receives: - Topic selection rejection history (last 5) ### Decision Criteria -When choosing between new / continue / combine: -- **New Topic**: When all existing topics are complete OR when a new mathematical avenue more aggressively attacks the user's WHOLE question as stated +When choosing between new / continue: +- **New Topic**: When all existing topics are complete OR when a new solution avenue more aggressively attacks the user's WHOLE question as stated - **Continue Existing**: When an incomplete brainstorm still attacks the whole prompt, or the next best necessary piece if a whole-question attack is absolutely not possible in one superintelligence brainstorm -- **Combine Topics**: When multiple existing brainstorms together produce a more direct rigorous route to the full prompt than keeping them separate ### Hard Code Guard: continue_existing on Completed Brainstorms The coordinator enforces a **hard code guard** in `_execute_topic_selection`: if the LLM selects `continue_existing` on a brainstorm whose status is `"complete"`, the action is **rejected** and the topic selection loop retries. This prevents runaway re-brainstorming on already-completed topics regardless of LLM judgment. The LLM prompt and validator prompt both instruct against this, but the code guard is the authoritative enforcement. -JSON schema and examples defined in `json-prompt-design.mdc`. Fields: `action` (new_topic/continue_existing/combine_topics), `topic_id`, `topic_ids`, `topic_prompt`, `reasoning`. +Production prompt/parser/model code is authoritative for the exact JSON contract. High-level fields: `action` (new_topic/continue_existing), `topic_id`, `topic_prompt`, `reasoning`. --- @@ -193,20 +203,18 @@ Receives the SAME context as the topic submitter: ### Validation Criteria **ACCEPT if**: -1. New topic addresses a genuinely valuable mathematical avenue not yet covered -2. Continue existing makes sense given the brainstorm's current state and mathematical depth -3. Combine topics is justified by genuine mathematical connections -4. Choice is relevant to the user's research goal -5. Reasoning is sound and mathematically grounded +1. New topic addresses a genuinely valuable solution avenue not yet covered +2. Continue existing makes sense given the brainstorm's current state and the depth appropriate to the objective +3. Choice is relevant to the user's research goal +4. Reasoning is sound and grounded under the rigor appropriate to its domain and claim types **REJECT if**: 1. New topic duplicates an existing brainstorm 2. Continue existing on a brainstorm that should be marked complete -3. Combine topics lacks clear mathematical justification -4. Choice ignores more valuable research avenues -5. Reasoning is flawed or lacks mathematical rigor +3. Choice ignores more valuable research avenues +4. Reasoning is flawed or lacks the rigor appropriate to its domain and claim types -JSON schema defined in `json-prompt-design.mdc`. Fields: `decision` (accept/reject), `reasoning`. +High-level fields: `decision` (accept/reject), `reasoning`; production prompt/parser/model code owns the exact contract. ### Rejection Feedback When rejected, the validator's reasoning is added to the topic submitter's rolling feedback cache (last 5 rejections), similar to aggregator submitter rejection logs. @@ -219,10 +227,10 @@ When rejected, the validator's reasoning is added to the topic submitter's rolli ### Purpose - The Crucial Mechanism for Compounding Knowledge -The autonomous research system's power comes from its ability to **compound knowledge across research cycles**. Each completed paper represents distilled mathematical insights that can inform and enhance future brainstorm explorations. +The autonomous research system's power comes from its ability to **compound knowledge across research cycles**. Each completed paper represents distilled mechanisms, evidence, designs, algorithms, mathematical results, proofs, or other solution insights that can inform and enhance future brainstorm explorations. By selecting reference papers BEFORE brainstorming begins, submitters can: -- Build upon promising mathematical frameworks from prior AI-generated papers while independently re-checking their claims +- Build upon promising mechanisms, evidence, designs, algorithms, mathematical frameworks, or proofs from prior AI-generated papers while independently re-checking their claims - Avoid re-exploring territory already covered in depth - Identify novel connections between new topics and previously explored results - Accelerate convergence on valuable insights by standing on prior work @@ -272,7 +280,7 @@ Context: Direct inject if fits (~40% budget), RAG if too large. No truncation. - Current brainstorm topic prompt (direct injection) - ALL Tier 2 paper titles + abstracts + outlines (direct injection if fits, RAG if too large) -JSON schemas defined in `json-prompt-design.mdc`. Two-step: submitter requests paper expansions (`expand_papers`, `proceed_without_references`), then makes final selection (`selected_papers`). Max 3 papers total for the topic cycle. +Two-step contract: the submitter requests paper expansions (`expand_papers`, `proceed_without_references`), then makes the final selection (`selected_papers`). Production prompt/parser/model code owns the exact contract. Maximum 3 papers total for the topic cycle. **Context Handling (DIRECT INJECTION FIRST, RAG SECOND):** @@ -303,8 +311,8 @@ JSON schemas defined in `json-prompt-design.mdc`. Two-step: submitter requests p Once a topic is validated and references selected, standard aggregation begins on that brainstorm topic. ### Architecture -- **3 Submitters**: Generate mathematical insights for the brainstorm topic -- **1 Validator**: Validates submissions (mathematical rigor, novelty, relevance) +- **3 Submitters**: Generate objective-relevant solution contributions for the brainstorm topic +- **1 Validator**: Validates submissions for domain- and claim-appropriate rigor, novelty, and relevance - **Pruning**: Cleanup review runs on the child Aggregator's run-local 7-acceptance cadence (same mechanism as Part 1) - **Same as Part 1 Aggregator**: Uses identical submitter/validator patterns, RAG cycling, etc. - **PARALLEL EXECUTION**: Submitters run in parallel regardless of boost status (boost is routing-only) @@ -432,7 +440,7 @@ Assess whether the current brainstorm has been sufficiently explored relative to - If CONTINUE_BRAINSTORM: Add feedback to rolling cache, return to aggregation - If WRITE_PAPER: Proceed to paper writing workflow (Tier 2) -JSON schemas defined in `json-prompt-design.mdc`. Completion submitter: `decision` (continue_brainstorm/write_paper), `reasoning`, `suggested_additions`. Self-validator: `validated` (bool), `reasoning`. +High-level contract: completion submitter uses `decision` (continue_brainstorm/write_paper), `reasoning`, and `suggested_additions`; self-validator uses `validated` (bool) and `reasoning`. Production prompt/parser/model code owns the exact contract. ### Completion Review Context - User's high-level research prompt @@ -473,7 +481,7 @@ Once completion review decides WRITE_PAPER (and self-validates), the system tran ### Two-Step Browsing Workflow (Additional Mode) -Same two-step browsing workflow as pre-brainstorm selection (expand request → final selection). JSON schemas defined in `json-prompt-design.mdc`. Already-selected papers shown as context; submitter requests expansion of remaining papers, then selects additional ones. Already-selected papers cannot be removed. +Same production two-step browsing contract as pre-brainstorm selection (expand request → final selection). Already-selected papers are shown as context; the submitter requests expansion of remaining papers, then selects additional ones. Already-selected papers cannot be removed. **Validator Role**: - Reference selection is handled by the dedicated reference selector workflow and hard caps @@ -517,7 +525,7 @@ Same two-step browsing workflow as pre-brainstorm selection (expand request → **Purpose**: Choose a title for the paper that will be compiled from this brainstorm. The selector sees 5 pre-validated candidate titles and may select one, synthesize, or propose a new title with justification. -JSON schema defined in `json-prompt-design.mdc`. Fields: `paper_title`, `reasoning`. +High-level fields: `paper_title`, `reasoning`; production prompt/parser/model code owns the exact contract. **DISTINCTION FOR TITLE VALIDATION:** @@ -744,12 +752,12 @@ When the child compiler completes the abstract phase and the parent wrapper obse - User's high-level research prompt - Instruction: "Review all papers and determine if ANY ONE paper should be REMOVED due to redundancy" -JSON schema defined in `json-prompt-design.mdc`. Fields: `should_remove` (bool), `paper_id` (or null), `reasoning`. +High-level fields: `should_remove` (bool), `paper_id` (or null), `reasoning`; production prompt/parser/model code owns the exact contract. 2. **Removal Criteria**: - Paper is REDUNDANT with other papers (content fully covered elsewhere) - Paper was MARGINALLY useful but other papers provide better coverage - - Paper's unique contributions are minimal compared to other papers + - Paper's unique solution mechanism, evidence, derivation, design, implementation, algorithm, theorem/proof, validation method, or risk analysis is minimal compared to other papers 4. **Conservative Approach**: - When in doubt, DO NOT remove @@ -796,6 +804,7 @@ Tier 3 synthesizes all accumulated research (Tier 2 papers) into a **final answe 2. **Independent Rejection Feedback**: Tier 3 has its own 10-rejection cache, separate from Tiers 1/2 3. **System Termination**: Once the final answer is complete, the autonomous system stops 4. **Modular Code Reuse**: Uses existing Tier 2 paper writing infrastructure (no duplicate code) +5. **Domain-Appropriate Synthesis**: Tier 3 presents the strongest defensible direct answer under the applicable rigor/evidence standard, distinguishes demonstrated results from proposals and required validation, and retains mathematics/formal proof as first-class modalities when relevant. ### Trigger Condition @@ -808,7 +817,7 @@ Tier 3 synthesizes all accumulated research (Tier 2 papers) into a **final answe ### Phase 1: Certainty Assessment -**Purpose**: Assess what can be answered WITH CERTAINTY from existing research papers. +**Purpose**: Assess the strongest defensible answer and the evidence status of its components from existing research papers. **Two-Step Paper Browsing** (same pattern as reference selection): @@ -824,13 +833,13 @@ Tier 3 synthesizes all accumulated research (Tier 2 papers) into a **final answe **Certainty Levels**: | Level | Description | |-------|-------------| -| `total_answer` | User's question can be FULLY answered with high confidence | -| `partial_answer` | Question can be partially answered with certainty | -| `no_answer_known` | Existing research doesn't provide an answer - MORE RESEARCH NEEDED | -| `appears_impossible` | The question appears mathematically impossible | +| `total_answer` | Complete answer is well-supported at the applicable standard | +| `partial_answer` | Only identified portions are well-supported | +| `no_answer_known` | Available papers do not support a defensible answer - MORE RESEARCH NEEDED | +| `appears_impossible` | Objective appears mathematically impossible, physically/constraint infeasible, internally inconsistent, or otherwise unsupported for a clearly reasoned reason | | `other` | Special cases | -JSON schema defined in `json-prompt-design.mdc`. Fields: `certainty_level` (total_answer/partial_answer/no_answer_known/appears_impossible/other), `known_certainties_summary`, `reasoning`. +High-level fields: `certainty_level` (total_answer/partial_answer/no_answer_known/appears_impossible/other), `known_certainties_summary`, `reasoning`; production prompt/parser/model code owns the exact contract. **Special Case - `no_answer_known`**: - If certainty assessment returns `no_answer_known`, Tier 3 exits @@ -848,15 +857,15 @@ JSON schema defined in `json-prompt-design.mdc`. Fields: `certainty_level` (tota | Format | Use Case | |--------|----------| | `short_form` | Answer can be presented coherently in single paper | -| `long_form` | Answer requires curated collection, multiple perspectives | +| `long_form` | Genuinely independent solution components require a curated collection | **Decision Factors**: - Complexity of user's question -- Number and diversity of relevant papers +- Independent mechanisms, evidence, implementations, proofs, validations, or risk analyses - Whether single coherent narrative is possible -- Whether papers naturally form a cohesive volume +- Whether the solution has a genuine multi-chapter dependency structure; paper count alone is insufficient -JSON schema defined in `json-prompt-design.mdc`. Fields: `answer_format` (short_form/long_form), `reasoning`. +High-level fields: `answer_format` (short_form/long_form), `reasoning`; production prompt/parser/model code owns the exact contract. **Validation**: Dedicated answer-format validation reviews the selection (10-rejection feedback loop) @@ -873,7 +882,7 @@ JSON schema defined in `json-prompt-design.mdc`. Fields: `answer_format` (short_ **Paper Title Requirement**: Title must DIRECTLY and TRANSPARENTLY answer the user's question. -JSON schema same as paper title selection (defined in `json-prompt-design.mdc`). Title must directly and transparently answer the user's question. +The production final-title contract follows paper title selection. The title must directly and transparently answer the user's question; production prompt/parser/model code owns the exact contract. ### Phase 3B: Long Form Answer (Volume) @@ -897,7 +906,7 @@ JSON schema same as paper title selection (defined in `json-prompt-design.mdc`). | `introduction` | Introduction paper (frames the volume, written LAST) | | `conclusion` | Conclusion paper (synthesizes findings, written second-to-last) | -JSON schema defined in `json-prompt-design.mdc`. Fields: `volume_title`, `chapters` (array with chapter_type/paper_id/title/order/description), `outline_complete`, `reasoning`. +High-level fields: `volume_title`, `chapters` (array with chapter_type/paper_id/title/order/description), `outline_complete`, `reasoning`; production prompt/parser/model code owns the exact contract. **Paper Writing Order (Long Form)**: 1. **Gap Papers** (if any): Write in chapter order @@ -1201,21 +1210,21 @@ Archive IDs are untrusted path components. Resolve `answer_id`, `paper_id`, and Runs automatically after every completed brainstorm (Tier 1) and completed Tier 2 paper-library paper, gated on `system_config.lean4_enabled`. Tier 3 short-form final papers may reuse the same paper-completion path; long-form volume chapters are saved as final-answer artifacts and do not currently run this checkpoint. When disabled, proof verification must not invoke Lean/proof model work or block brainstorm/paper completion. -**Proof Framing Gate (one-shot, at autonomous start)**: On fresh autonomous starts, the coordinator runs `_run_proof_framing_gate()` before research begins. A single LLM call on the user prompt decides `is_proof_amenable` (`build_proof_framing_gate_prompt` → `autonomous_proof_framing_gate` role). The gate errs on the side of `true` when formal proof can help the user's prompt — it returns `false` when the prompt is purely empirical, engineering-focused, or has no meaningful prompt-relevant mathematical content. If `true`, `PROOF_FRAMING_CONTEXT` (which directs submissions to pursue theorems/lemmas/formalizations that directly answer, support, or advance the user prompt, with novelty/non-triviality valuable only inside that boundary) is appended to every subsequent submitter prompt via `_append_proof_framing()` and persisted to workflow state for crash recovery. Decision is broadcast via `proof_framing_decided`. Lean/proof execution remains separately gated by `lean4_enabled`. +**Proof Framing Gate (one-shot, at autonomous start; essential when proof output is allowed)**: On fresh autonomous starts with Mathematical Proofs allowed, the coordinator runs `_run_proof_framing_gate()` before research begins. When Mathematical Proofs is disabled in Allowed Outputs, the framing model call and all proof-specific prompt context—including framing, direct proof-database context, failed-proof hints, known-proof summaries, and Assistant proof-memory packs for Autonomous and its child Aggregator/Compiler roles—are skipped, including on resume. Otherwise this LLM proof-emphasis gateway, its eager true bias, persistence/resume, and later prompt injection must not be bypassed, deferred, narrowed, or weakened. A single LLM call on the user prompt decides `is_proof_amenable` (`build_proof_framing_gate_prompt` → `autonomous_proof_framing_gate` role). The gate errs on the side of `true` when formal proof can help the user's prompt — it returns `false` when the prompt is purely empirical, engineering-focused, or has no meaningful prompt-relevant mathematical content. If `true`, `PROOF_FRAMING_CONTEXT` (which directs submissions to pursue theorems/lemmas/formalizations that directly answer, support, or advance the user prompt, with novelty/non-triviality valuable only inside that boundary) is appended to every subsequent submitter prompt via `_append_proof_framing()` and persisted to workflow state for crash recovery. Decision is broadcast via `proof_framing_decided`. Lean/proof execution remains separately gated by `lean4_enabled`, and the complete enabled Lean pipeline remains protected. **Automatic proof rounds**: Proofs-only autonomous runs let brainstorm proof checkpoints run the pipeline below for up to 4 rounds. Round 1 uses normal candidate identification. After each completed round, the coordinator asks again whether there are any remaining proofs in the same source that directly solve the user's prompt or substantially advance a solution path, with newly verified proofs visible through the proof-library context. The checkpoint stops early when a round finds no candidates. When research papers and mathematical proofs are both enabled, automatic proof checkpoints after brainstorming and after completed paper writing are single-round so paper generation remains the main output path. Manual proof checks, deferred paper retry checks, compiler rigor, and LeanOJ are single-round unless their own mode rules say otherwise. The per-source reservation spans the whole automatic multi-round checkpoint, and completed earlier rounds must never overwrite an active later-round resume cursor. -**Assistant memory-support role**: When Session History Memory is enabled, each active workflow uses one shared Assistant memory role (`autonomous_assistant` for autonomous, `leanoj_assistant` for LeanOJ), not one Assistant per submitter/proof candidate/solver lane. Assistant runs beside eligible non-validator, non-critique brainstorming, selection, writing, proof, and Tier 3 roles as optional verified proof-memory support; short-lived topic/title exploration candidate mini-aggregators are excluded. It may provide up to 7 prior verified proof supports from enabled proof-history corpora and reuse useful packs for two eligible receiver reads before refresh, but validators never receive Assistant context and parent workflows never wait on Assistant. Assistant skips true no-external-history targets because it only performs proof-memory retrieval for now. Durable cooldown is keyed to stable run scope, grouping transient task IDs/roles while preserving real source/session separation; zero-useful retrieval can eventually shut down for the run, while stagnant same-pack retrieval backs off without shutdown. User live activity should show normal Assistant retrieval result summaries and explicit Assistant model-output failures, not skip/backoff/shutdown turns. Stale live packs/state clear only on explicit reset/clear or Session History Memory disable. +**Assistant memory-support role**: When Session History Memory is enabled, each active workflow uses one shared Assistant memory role (`autonomous_assistant` for autonomous, `leanoj_assistant` for LeanOJ), not one Assistant per submitter/proof candidate/solver lane. Assistant runs beside eligible non-validator, non-critique brainstorming, selection, writing, proof, and Tier 3 roles as optional verified Lean proof-history support; it cannot redirect or mathematically reinterpret an unrelated parent goal, and returning no support is valid. Short-lived topic/title exploration candidate mini-aggregators are excluded. It may provide up to 7 prior verified proof supports from enabled proof-history corpora and reuse useful packs for two eligible receiver reads before refresh, but validators never receive Assistant context and parent workflows never wait on Assistant. Lean 4 formalization attempts may reuse the last available Assistant proof pack, but they must not refresh Assistant retrieval until the Lean attempt phase has finished. Assistant skips true no-external-history targets because it only performs proof-memory retrieval for now. Durable cooldown is keyed to stable run scope, grouping transient task IDs/roles while preserving real source/session separation; zero-useful retrieval can eventually shut down for the run, while stagnant same-pack retrieval backs off without shutdown. User live activity should show normal Assistant retrieval result summaries and explicit Assistant model-output failures, not skip/backoff/shutdown turns. Top-level Stop clears live/latest packs; schema-compatible SQLite cache state may remain for re-filtered reuse, while explicit Clear/reset or Session History Memory disable clears durable run-scoped state. **Pipeline** (`backend/autonomous/core/proof_verification_stage.py`): 1. **Candidate identification** — `ProofIdentificationAgent` (`build_proof_identification_prompt`) extracts prompt-relevant, high-impact new/novel theorem candidates from brainstorm or paper content. For brainstorms, the brainstorm topic is bounded source-local metadata only and must not broaden eligibility beyond proofs that directly solve, or visibly build toward solving, the user prompt. This stage is not a general known-knowledge-base builder: candidates are ordered by impact on the user prompt (direct solutions/impossibility results first, then decisive reductions, obstructions, and structural theorems that themselves make major progress). Candidate JSON includes non-empty `expected_novelty_tier`, `prompt_relevance_rationale`, `novelty_rationale`, and `why_not_standard_known_result`; invalid, missing-rationale, or `not_novel` candidates are skipped before Lean 4 cost is incurred. Routine helpers, supporting lemmas, trivial/easy proofs, standard/textbook/Mathlib restatements, off-prompt curiosities, program-local firsts, minor reformulations, and single-tactic/routine proof goals are rejected even as fallback targets. 2. **Optional Mathlib lemma search** — `MathlibLemmaSearchAgent` surfaces relevant existing lemmas into the formalization prompt, tied to the target theorem, user prompt, and brainstorm topic when present 3. **Optional SMT early-exit** — when `smt_enabled`, `SmtClient` classifies candidates conservatively; only valid `unsat` checks become Lean tactic hints (for example `nativeDecide`, `omega`, `decide`, `norm_num`, `linarith`, or `polyrith`-style hints). `sat`, `unknown`, failed translation, and non-amenable candidates produce no hint. SMT results are never stored as standalone proofs; prompts receive the same user prompt + brainstorm topic relevance context -4. **Lean 4 formalization attempts** — two-phase retry: up to 3 full-proof attempts via `ProofFormalizationAgent.prove_candidate`, then up to 2 multi-tactic script attempts via `prove_candidate_tactic_script` (5 total per candidate). Formalization prompts receive the same source title/brainstorm topic context plus the complete source brainstorm/paper as mandatory direct context; focused excerpts are supplemental only. The latest Assistant memory pack may be injected as optional proof-pattern/dependency guidance, dropping it before mandatory source context if the prompt would overflow. Direct `search_lean_proofs` prefetch/tool loops are explicit legacy/debug or narrow emergency-repair paths only, not the normal retrieval path. When full SyntheticLib4 Lean code is injected into a prompt and the formalization succeeds, MOTO records a local non-whole-code `model_visible_context` usage attestation; live whole-code usage submission remains tied to the future SyntheticLib4 service contract. Prompt source reads strip appended generated-proof sections so verified proofs enter through the explicit proof-library context, while canonical proof files and user-visible appendices remain preserved. If the complete source cannot fit, fail visibly instead of silently truncating or proving from excerpt-only context. Prior `FailedProofCandidate` failure hints from `proof_database.inject_failure_hints_into_prompt()` thread into each retry. -5. **Post-Lean preservation + novelty check** — hard integrity rejects only fake proof devices such as new `axiom`/`constant`/`opaque`; statement mismatch is classified as a downshift, never a discard. `autonomous_proof_novelty` ranks the actual Lean-verified theorem against standard references/Mathlib, prompt relevance, and the proof library; program-local firsts are not novel. -6. **Storage** — `proof_registration.register_verified_lean_proof()` uses `proof_database.add_proof_if_absent()` to persist novel and known proofs as session-aware records (`proofs_index.json`, `proof_.json`, `proof__lean.lean`). Dependency extraction runs after initial registration and patches the stored record afterward; `proof_verified` may therefore emit before dependency metadata is attached. Same-source duplicate detection is scoped to source type/id + normalized theorem statement + normalized Lean code and must return `duplicate=True` to callers so source files are not appended twice. Cross-source exact theorem/code matches are stored as non-novel reused proof occurrences without spending a novelty API call. If a Lean-accepted proof nevertheless proves a different or narrower theorem than the intended candidate, store the actual theorem statement and retain the original target in notes; this is preservation of a real artifact, not permission to target supporting lemmas. If the active/current `proofs_index.json` is corrupt, rebuild from existing `proof_*.json` record files instead of replacing the library with an empty index; cross-session library scans may skip unreadable historical indexes unless explicitly rebuilt. Automatic proof checkpoints append source-file proof sections only for non-duplicate novel records. User-triggered "Try to Prove This" / manual proof checks append any non-duplicate Lean-verified proof to the target brainstorm or paper source appendix regardless of novelty rating; preserving known verified proofs here is intentional RALPH-loop state so users can inspect validated work and later checks avoid redoing exact proofs that ranked `not_novel`. Cross-session read access is provided by `proof_database.list_proof_library()` (all sessions, novelty-filtered) and `proof_database.get_library_proof(session_id, proof_id)`, consumed by the `ProofLibrary` UI component and `/api/proofs/library` endpoints. +4. **Lean 4 formalization attempts** — two-phase retry: up to 3 full-proof attempts via `ProofFormalizationAgent.prove_candidate`, then up to 2 multi-tactic script attempts via `prove_candidate_tactic_script` (5 total per candidate). Formalization prompts receive the same source title/brainstorm topic context plus the complete source brainstorm/paper as mandatory direct context; focused excerpts are supplemental only. The last available Assistant memory pack may be injected as optional proof-pattern/dependency guidance, dropping it before mandatory source context if the prompt would overflow; formalization attempts reuse that pack without scheduling Assistant refreshes from Lean feedback. Direct `search_lean_proofs` prefetch/tool loops are explicit legacy/debug or narrow emergency-repair paths only, not the normal retrieval path. Provider max-output truncation before usable Lean code is a failed attempt with live activity, not a research-stopping provider failure. When full SyntheticLib4 Lean code is injected into a prompt and the formalization succeeds, MOTO records a local non-whole-code `model_visible_context` usage attestation; live whole-code usage submission remains tied to the future SyntheticLib4 service contract. Prompt source reads strip appended generated-proof sections so verified proofs enter through the explicit proof-library context, while canonical proof files and user-visible appendices remain preserved. If the complete source cannot fit, fail visibly instead of silently truncating or proving from excerpt-only context. Prior `FailedProofCandidate` failure hints from `proof_database.inject_failure_hints_into_prompt()` thread into each retry. +5. **Post-Lean preservation + novelty check** — hard integrity rejects only fake proof devices such as new `axiom`/`constant`/`opaque`; statement mismatch is classified as a downshift, never a discard. `autonomous_proof_novelty` ranks the actual Lean-verified theorem against public/standard references, Mathlib, and current prompt relevance without private proof-history input; program-local firsts are not novel. Novelty classifier malformed/schema-invalid JSON gets one bounded sanitized retry before falling back to `not_novel`. +6. **Storage** — Every newly Lean-verified current-run occurrence gets a full record and independent novelty pass even when private history has identical theorem/code. Afterward, enabled historical MOTO/manual/LeanOJ corpora are silently checked by versioned canonical theorem+Lean identity with the current run/session excluded. A prompt-novel cross-run exact match retains its independent tier/reasoning as provenance and receives the searchable `duplicate_novel` storage/display overlay; `not_novel` remains unchanged, and no duplicate suppresses current-run context or persistence. Persist canonical user prompts and stable run IDs; SyntheticLib4 integrity hashes remain separate from internal canonical hashes. -**Unified proof search**: Assistant is the preferred owner of routine workflow-memory retrieval. Unified proof search retrieves bounded, provenance-carrying, fully Lean-verified examples from prior MOTO proof history, LeanOJ verified artifacts, manual proof history, and authorized SyntheticLib4 snapshots while excluding active/current-run proof records; the configured Assistant LLM then selects/refines at most 7 model-visible supports. Do not add a second model-rater role separate from Assistant, and do not publish live support packs without configured Assistant LLM selection. Search is not candidate discovery, not a workflow stop condition, and not a replacement for MOTO Lean/integrity/registration gates; SyntheticLib4/search unavailability should return structured warnings while local workflows continue where possible. +**Unified proof search**: Assistant owns routine workflow-memory retrieval from prior MOTO, LeanOJ, manual, and authorized SyntheticLib4 history while excluding the requesting run/session from fresh and reused packs. Exact neighborhoods use `moto-proof-identity-v1` and composite `(corpus, run_id, session_id, source_type, source_id)` occurrence identity. Derived SQLite schema v2 stores canonical columns and atomically publishes a sibling rebuild when schema/identity compatibility fails. SyntheticLib4 publisher hashes remain integrity data. Assistant selects at most 7 supports; search never changes candidate eligibility or bypasses proof gates. **Parallelism (two-phase execution per stage run)**: Steps 2–4 above (the per-candidate "Phase A" pipeline: lemma search → optional SMT hint → `prove_candidate` → `prove_candidate_tactic_script` → `proof_attempts_exhausted` broadcast on failure) run concurrently across identified candidates inside a single `ProofVerificationStage.run()` invocation. `system_config.proof_max_parallel_candidates` controls Phase A batching: the default is `6`; `0` (env: `MOTO_PROOF_MAX_PARALLEL_CANDIDATES` / `PROOF_MAX_PARALLEL_CANDIDATES`) means unlimited; positive values run strict batches of that size before the next batch starts. Phase A parallelizes agent/model work, but actual Lean 4 subprocess verification is serialized by `Lean4Client` behind a shared execution lock so all candidates queue one-at-a-time against the shared Mathlib workspace; LSP mode remains independently serialized by its operation lock and subprocess fallback uses the same shared queue. The identification stage (step 1) filters off-prompt, trivial, and well-known results before Phase A begins, so Phase A only processes prompt-relevant theorem candidates. Completed candidates are consumed as tasks finish, and steps 5–6 (the "Phase B" post-processing: novelty assessment, `add_proof`, dependency extraction via `ProofDependencyExtractor`, `append_proofs_section`, `novel_proof_discovered` / `known_proof_verified` broadcast, `record_failed_candidate` for brainstorm failures) are performed strictly **one-at-a-time** in Phase-A completion order inside that driver loop so later candidates can observe earlier stored proofs as MOTO dependencies. If any Phase-A task raises account-credit exhaustion, the driver cancels siblings and re-raises so the autonomous coordinator persists the proof checkpoint pause and waits for OpenRouter reset before retrying. Other Phase-A exceptions cancel siblings, save an error checkpoint, broadcast completion with error state, and return `had_error=True` so the coordinator recovery path can continue without orphaned background API calls. `should_stop` is plumbed into each Phase-A pipeline and checked before each Phase-B pass, so a stop-request short-circuits cleanly without leaking tasks. @@ -1231,7 +1240,7 @@ Runs automatically after every completed brainstorm (Tier 1) and completed Tier **Proof runtime config snapshot** (`research_metadata.set_proof_runtime_config`): Captures a `ProofRuntimeConfigSnapshot` with three `ProofRoleConfigSnapshot` entries — `brainstorm` (from first aggregator submitter config), `paper` (from Rigor & Proofs Submitter config for proof work), `validator` (from validator config). Each holds provider, model_id, openrouter_provider, openrouter_reasoning_effort, lm_studio_fallback_id, context_window, max_output_tokens, and supercharge_enabled. Lets manual checks run without an active autonomous session when a request snapshot is not supplied. -**Proof WebSocket events** are broadcast through the standard `/api/ws` stream for user-visible progress. Do not make every internal progress notification a rule-level invariant, but keep frontend-consumed events stable and update the hosted contract/API version when changing them. Autonomous proof-round progress events include `proof_round_index` and `proof_max_rounds`. `proof_verified` is emitted only after the proof has passed integrity checks and has been registered/reused in the proof database; payloads include `proof_id`. Novel/known/duplicate proof registration events include the validator's `novelty_tier` and `novelty_reasoning` so live activity can show whether the Lean 4 proof was rated novel or not novel. +**Proof WebSocket events** are broadcast through the standard `/api/ws` stream for user-visible progress. Keep frontend-consumed events stable and bump the hosted contract when changing them. `proof_verified` emits only after registration/reuse and includes `proof_id`; novelty events expose the stored display tier/reasoning, while independent novelty and duplicate provenance live in typed proof records. **Proof Stage Critical Invariants**: 1. Proof stage is skipped when `lean4_enabled=False`; it must not run Lean/proof model work or block brainstorm or paper completion @@ -1239,19 +1248,19 @@ Runs automatically after every completed brainstorm (Tier 1) and completed Tier 3. Subprocess checker must continue to work when `lean4_lsp_enabled=False`; LSP path must not regress subprocess behavior when enabled. Missing/corrupt `.olean` cache errors must trigger one workspace repair/retry, then fail the current check with `LEAN 4 WORKSPACE ERROR` if repair fails; these infrastructure failures must not fall through into tactic mode, emit `proof_attempts_exhausted`, or burn all proof attempts as ordinary Lean feedback 4. Proof storage is session-aware (`session_manager` → `get_proofs_dir()`) and falls back to the legacy `backend/data/proofs/` layout when no session is active 5. Per-source reservation lock prevents concurrent proof checks on the same `{source_type}:{source_id}` (autonomous vs manual interleaving) -6. Novel proofs become highest-priority direct-injection context for subsequent brainstorm/paper submitters via `proof_database.inject_into_prompt()` and stored `ProofRecord` summaries; `inject_failure_hints_into_prompt()` is reserved for unresolved failed proof targets +6. Only independently prompt-novel, non-duplicate display tiers become highest-priority direct-injection context via `proof_database.inject_into_prompt()`; `duplicate_novel` and `not_novel` remain searchable but excluded there. Failure hints are limited to unresolved active/resumable targets, and clear/archive removes failed candidates while preserving verified history. 7. Proof certificates stay text-based (`.lean` source + JSON metadata) — no binary artifacts 8. Hosted/generic mode keeps `lean4_enabled` and `smt_enabled` default false and the hosted image stays Lean-free and Z3-free (no proof binaries in the `python:3.12-slim` runtime) -9. Proof framing gate runs once on fresh autonomous starts; the resulting `proof_framing_active` flag and `PROOF_FRAMING_CONTEXT` are persisted in workflow state for crash recovery. Lean/proof model execution remains controlled by `lean4_enabled`. +9. When Mathematical Proofs is allowed, the proof framing gate runs once on fresh autonomous starts; the resulting `proof_framing_active` flag and `PROOF_FRAMING_CONTEXT` are persisted in workflow state for crash recovery. When proof output is disabled, the gate and framing injection are skipped on fresh starts and resume. Lean/proof model execution remains controlled by `lean4_enabled`. 10. Candidate identification (`build_proof_identification_prompt`) is an impact-first user-prompt relevance gate, not a known-knowledge-base builder. It rejects off-prompt curiosities, routine helper lemmas, supporting lemmas, trivial/easy proofs, standard/textbook/Mathlib restatements, program-local firsts, minor reformulations/local formalizations, and single-tactic/routine proof goals, then returns candidates ordered by direct impact on the user's prompt: direct solutions or impossibility results first, then decisive reductions, obstructions, and structural theorems that themselves make major progress on the requested problem. Candidate prompts require expected novelty plus non-empty prompt-relevance, novelty, and anti-standard-result rationale fields; invalid, missing-rationale, or `not_novel` candidates are skipped before Lean cost. Every candidate that passes this gate is attempted — `proof_max_parallel_candidates` defaults to 6, `0` runs all Phase A work without a batch cap, and positive values run strict batches without truncating the post-identification candidate list; actual Lean 4 subprocess verification queues one-at-a-time through `Lean4Client`, and Phase B (novelty / `add_proof` / dependency extraction / brainstorm+paper `append_proofs_section` / novel/known broadcasts / `record_failed_candidate`) remains strictly serialized in Phase-A completion order so intra-batch MOTO dependencies and per-source proof appending stay coherent -10b. Proof-identification transport, empty-output, malformed/schema-invalid JSON, or no-JSON reasoning-only failures must preserve the proof checkpoint as an error; only a valid JSON no-candidates response may advance past proof checking as no candidates. +10b. Proof-identification empty-output, provider output truncation, malformed/schema-invalid JSON, or no-JSON reasoning-only failures get one bounded sanitized JSON retry; if recovery still fails, preserve the proof checkpoint as an error. Only a valid JSON no-candidates response may advance past proof checking as no candidates. 11. Each Phase-A task owns its own `ProofIdentificationAgent` / `MathlibLemmaSearchAgent` / `ProofFormalizationAgent` instance to keep per-agent `task_sequence` counters collision-free; account-credit exhaustion cancels siblings and preserves the checkpoint for provider pause/retry, while other Phase-A failures cancel siblings and return a structured `had_error=True` stage result for coordinator recovery 12. `should_stop` propagates into Phase A and is re-checked before each Phase-B pass so stop-requests short-circuit without leaking tasks or partially-applied Phase-B writes. Autonomous proof checkpoints persist the resolved candidate cursor, processed candidate IDs, proof labels/indexes, Lean attempt feedback, and post-Lean metadata needed for Phase B (including accepted theorem names/code) in workflow state so provider-credit pause, Stop/Start, restart, and model changes resume remaining candidates instead of re-identifying from Proof A; provider pauses during Phase A or Phase B must preserve the same checkpoint. Proof checkpoint completion markers are source- and round-trigger-scoped and must not transfer between brainstorms/papers or overwrite active later-round cursors. 13. Compiler rigor mode (`submit_rigor_lean_theorem`, `_rigor_loop`) is NOT parallelized and is capped at 5 consecutive cycles per rigor loop — rigor cycles discover, verify, and route one theorem per cycle (inline for eligible existing-paper claims, appendix-only for extension-derived theorems or placement fallback) so each verified theorem lands in the paper before the next discovery; the parallel candidate pipeline lives only in `ProofVerificationStage` 14. Post-Lean integrity scanning rejects newly introduced `axiom`, `constant`, and `opaque` declarations even when the declaration name appears on following lines. Generated source text is not an authorization baseline unless explicitly passed as allowed baseline. 15. Lean-accepted real proof code is preservation-worthy even when it misses the intended candidate. Alignment classifiers may downshift the stored theorem statement and ranking may classify it as `not_novel`, but the proof artifact must not be discarded except for hard integrity failures. 16. Unified proof/history search is bounded retrieval support only. It must never broaden proof candidate identification beyond the user's prompt, replace mandatory source context, or count SyntheticLib4 retrieved records as new MOTO proofs without the normal MOTO artifact path. -17. Assistant memory-support retrieval is a parallel, non-blocking configured LLM role across eligible non-validator, non-critique brainstorming, writing, proof, selection/path, and final-answer roles. It provides optional up-to-7 verified proof supports, reuses useful packs for two eligible receiver reads before refresh, skips no-external-history targets, uses durable run-scoped cooldown for repeated zero-useful or stagnant retrieval, shuts down only for repeated zero-useful retrieval in the current run, hides skip/backoff/shutdown turns from live activity while showing Assistant model-output failures, clears stale live packs/state on explicit reset/clear or Session History Memory disable, and never alters phase transitions or candidate eligibility. +17. Assistant memory-support retrieval is a parallel, non-blocking configured LLM role across eligible non-validator, non-critique brainstorming, writing, proof, selection/path, and final-answer roles. It provides optional up-to-7 verified supports and reuses useful packs for two eligible receiver reads before refresh, but every cached/latest/stale/OAuth-cooldown reuse is re-filtered against the requesting stable run/session; an emptied pack falls through to normal retrieval. Durable run-scoped cooldown, hidden skip/backoff/shutdown activity, explicit failure visibility, clear/reset semantics, and phase/candidate neutrality remain unchanged. --- diff --git a/.cursor/rules/program-directory-and-file-definitions.mdc b/.cursor/rules/program-directory-and-file-definitions.mdc index 535133d..0f68ee0 100644 --- a/.cursor/rules/program-directory-and-file-definitions.mdc +++ b/.cursor/rules/program-directory-and-file-definitions.mdc @@ -12,6 +12,8 @@ LM Studio and its pre-loaded models can be reached at "http://127.0.0.1:1234" (o `backend/data/` and `backend/logs/` are the **default desktop roots**, not immutable global paths. - The active backend instance may override mutable roots with `MOTO_DATA_ROOT` and `MOTO_LOG_ROOT` +- Exactly one backend process may own a data root at a time; an OS-backed lifespan lease is authoritative even for direct launches and explicit overrides. Chroma is derived cache state: Windows desktop replaces prior-process native state before first open, while hosted Linux retains persistence. +- Runtime root changes after import must use the atomic `bind_runtime_roots()` contract, which re-derives the complete mutable path graph; long-lived stores resolve against the active root and RAG opens Chroma lazily. - Secret persistence may be isolated per instance with `MOTO_SECRET_NAMESPACE` - Frontend browser persistence may be isolated on shared origins with `VITE_MOTO_STORAGE_PREFIX`; launch/control-plane config may supply `MOTO_FRONTEND_STORAGE_PREFIX` and project it into the frontend env - Hosted protected request size may be capped with `MOTO_GENERIC_MAX_REQUEST_BYTES` / `GENERIC_MAX_REQUEST_BYTES` (default 16 MiB) @@ -44,7 +46,9 @@ project-root/ │ │ ├── boost_manager.py # Singleton boost manager (tracks boost modes: next-count, always-prefer, category; aliases absorbed LeanOJ path-decision tasks into Final Solver boost category) │ │ ├── boost_logger.py # Boost API call logger (persists redacted/default-safe entries to boost_api_log.txt) │ │ ├── workflow_predictor.py # Legacy/shared workflow prediction helper; active coordinators maintain their own workflow_tasks -│ │ ├── workflow_start_guard.py # Process-wide async guard for atomic mutually-exclusive workflow starts +│ │ ├── workflow_start_guard.py # Process-wide async guard and committed owner for atomic mutually-exclusive workflow starts +│ │ ├── runtime_root_lock.py # Cross-process OS-backed lease enforcing one backend owner per active data root +│ │ ├── sleep_inhibitor.py # Owner-based Windows desktop idle-sleep inhibitor for active top-level workflows │ │ ├── free_model_manager.py # Free model rotation/cooldown singleton (looping + auto-selector backup) │ │ ├── model_error_utils.py # Shared helpers for non-retryable provider/config failures; only recoverable credit exhaustion should pause workflows │ │ ├── provider_pause.py # Process-local provider-credit pause/resume signal used by proof workflows and OpenRouter reset @@ -61,8 +65,10 @@ project-root/ │ │ ├── provider_notification_store.py # Non-secret recent provider/OAuth failure notifications for missed-popup recovery │ │ ├── build_info.py # Build identity resolver (manifest + git HEAD/ZIP stamp + env overrides) │ │ ├── path_safety.py # Safe path resolution helpers (realpath/normpath containment checks) -│ │ ├── syntheticlib4_client.py # Contract-first SyntheticLib4 mock/offline client for release/status/retrieve/account-proof work, API-key secret scaffolding, snapshot metadata validation, and built-in fallback records when test fixtures are absent -│ │ ├── proof_search/ # Unified proof-search models, source normalizers, batched SQLite/FTS indexer, freshness/toggle-aware service, `search_lean_proofs` tool adapter, and Assistant memory-pack coordinator/ranker/cache +│ │ ├── syntheticlib4_client.py # Contract-first SyntheticLib4 local-snapshot client for release/status/retrieve/account-proof work and API-key scaffolding; inactive runtime has zero records, while explicit fixtures remain test-only +│ │ ├── proof_identity.py # `moto-proof-identity-v1`: canonical theorem whitespace + Lean newline/outer-whitespace hashes, separate from publisher integrity hashes +│ │ ├── proof_search/ # Typed records/normalizers; SQLite schema v2 atomic index; composite exact neighborhoods; freshness/hydration/tool service; policy-versioned at-most-64→21→7 run-safe Assistant selection +│ │ ├── solution_path/ # Typed run-scoped Progressive Solution Path state, durable serial proposal review engine, and prompt integration helpers │ │ ├── fastembed_provider.py # FastEmbed embedding wrapper (generic mode only, lazy-imported) │ │ ├── lean4_client.py # Lean 4 proof checker client (subprocess gated on `lean4_enabled`, optional LSP persistent mode gated on `lean4_lsp_enabled`; offloads temp/workspace filesystem operations from the FastAPI event loop) │ │ ├── lean_proof_integrity.py # Shared post-Lean integrity gate (rejects fake axiom/constant/opaque devices and validates theorem-statement alignment) @@ -139,7 +145,7 @@ project-root/ │ │ │ └── proof_dependency_extractor.py # Parses verified Lean 4 code to extract `ProofDependency` records (imports, Mathlib lemmas, MOTO-origin refs) │ │ ├── agents/ │ │ │ ├── __init__.py # Package initialization -│ │ │ ├── topic_selector.py # Topic selection submitter (new/continue/combine) +│ │ │ ├── topic_selector.py # Topic selection submitter (new/continue) │ │ │ ├── topic_validator.py # Topic selection validator │ │ │ ├── completion_reviewer.py # Brainstorm completion review (SPECIAL SELF-VALIDATION) │ │ │ ├── reference_selector.py # Reference paper selection workflow @@ -204,7 +210,7 @@ project-root/ │ │ ├── autonomous.py # Autonomous Research API endpoints │ │ ├── leanoj.py # LeanOJ Proof Solver API endpoints (`/api/leanoj/*`: start with matching-progress resume, stop, status, master-proof draft/edit summaries, current-run proofs, cross-session library, skip-brainstorm, force-brainstorm, clear) │ │ ├── boost.py # Boost API endpoints (enable/disable/toggle/status + OpenRouter provider endpoint metadata) -│ │ ├── workflow.py # Workflow API endpoints (predictions/history) +│ │ ├── workflow.py # Workflow predictions plus typed Progressive Solution Path snapshot and generation-fenced repair-resume endpoints │ │ ├── update.py # Update/check endpoints for launcher/updater state (`POST /api/update/pull`, `GET /api/update/pull-status`) │ │ ├── download.py # PDF generation endpoint via Playwright (desktop only; sanitize/block external requests; returns 501 in generic mode) │ │ ├── openrouter.py # OpenRouter API endpoints (global key, models/providers via header/body keys only, LM Studio availability, model cache, reset exhaustion) @@ -212,7 +218,7 @@ project-root/ │ │ ├── websocket.py # WebSocket for real-time updates (generic proxy auth or desktop one-time tickets before accept) │ │ ├── features.py # GET /api/features — shared build identity plus stable capability flags; GET /api/update-notice — launcher/runtime-refreshed update notice │ │ ├── proofs.py # Proof database + Lean 4/SMT runtime + manual proof-check + certificate export + dependency graph routes; current proof listing/detail/certificate/graph routes accept `scope=autonomous|manual` so active manual writer proofs use their own instance-level store; proof-library routes accept `scope=autonomous|manual`, with manual scope reading archived manual proof runs only; listing proofs (`GET /`, `/novel`, `/known`, `/library*`) and certificate/lean downloads (`/{id}/certificate`, `/{id}/certificate.lean`) are always available regardless of `lean4_enabled`; dependency/graph routes and `/check` are gated on `lean4_enabled`; `/status` uses short timeouts so it never blocks the UI -│ │ ├── proof_search.py # Unified proof-search routes (`/api/proof-search/*`) over canonical MOTO proofs and SyntheticLib4 fixture/snapshot records; public search is capped at 7 combined results, overview is filtered by enabled corpus toggles, and detail hydration is bounded +│ │ ├── proof_search.py # Unified overview/search/reindex, bounded detail hydration (`session_id`/`search_id`/`run_id`), latest Assistant pack, and paginated support-lineage routes; public search caps at 7 │ │ ├── syntheticlib4.py # SyntheticLib4 corpus status/release/refresh/safe local import/reindex/retrieve-batch/account-proof routes plus production OAuth placeholders; public retrieve-batch is capped at 7 records │ │ ├── connectivity.py # Non-secret grouped connectivity status/toggle routes for inference providers and optional skills │ │ └── health.py # GET /api/health — readiness/liveness probe with slim instance/build metadata @@ -264,7 +270,7 @@ project-root/ │ │ ├── proofs/ # Legacy (non-session) Lean 4 proof storage (mirrors per-session proofs/ layout) │ │ ├── manual_proofs/ # Active manual Aggregator/Compiler proof storage; archived/cleared on manual run reset so old proofs do not enter new manual prompts │ │ ├── manual_proof_runs/ # Archived manual proof runs for history/library viewing only, never active prompt context -│ │ ├── proof_search/ # Generated unified proof-search SQLite/FTS index under the active data root +│ │ ├── proof_search/ # Derived schema/identity-versioned SQLite/FTS index, atomically rebuildable from canonical proof sources │ │ ├── syntheticlib4/ # Planned authorized SyntheticLib4 local snapshots/status/cache under the active data root │ │ ├── leanoj_sessions/ # LeanOJ run state (state.json, master_proof.lean, master_proof_edits.jsonl, master_proof_snapshots.jsonl, phase counters, proof fragments, attempts, verified final Lean code; stop/crash resumes unless cleared) │ │ ├── leanoj_partial_proofs/ # LeanOJ partial/supporting proof scaffold JSONL store, keyed by session @@ -306,12 +312,12 @@ project-root/ │ │ │ ├── FinalAnswerLibrary.css # Final answer library styles │ │ │ ├── ArchiveViewerModal.jsx # Research lineage archive viewer (papers + brainstorms) │ │ │ ├── ArchiveViewerModal.css # Archive viewer styles -│ │ │ ├── MathematicalProofs.jsx # Live-session proof tab reused by Autonomous and Manual modes (proof lists/status, manual checks, dependency graph, certificate/Lean exports via scoped `/api/proofs`; proof events are filtered by scope) +│ │ │ ├── MathematicalProofs.jsx # Flat live-session per-proof tab reused by Autonomous and Manual modes (no prompt grouping/display; full proof content; status, manual checks, dependency graph, certificate/Lean exports via scoped `/api/proofs`; proof events are filtered by scope) │ │ │ ├── MathematicalProofs.css # Proof library styles │ │ │ ├── ProofGraph.jsx # Proof dependency graph view (hand-rolled SVG; Build 4, may escalate to reactflow in Build 5 if needed) │ │ │ ├── ProofGraph.css # Proof graph styles │ │ │ ├── ProofNotificationStack.jsx # Persistent popup notifications for novel proof discoveries -│ │ │ ├── ProofLibrary.jsx # Cross-session proof library viewer with novelty filter (all sessions, grouped by research run; sub-tab inside CompletedWorksLibrary; fetches via `/api/proofs/library`) +│ │ │ ├── ProofLibrary.jsx # Historical proof library with novelty filter, collapsed run/prompt groups, bounded prompt-header previews, and full expanded proof content; used by Autonomous Completed Works and separate Manual Completed Proof Works │ │ │ ├── ProofLibrary.css # Proof library viewer styles │ │ │ ├── Stage2PaperHistory.jsx # Tier 2 paper history list (grouped per research run; sub-tab inside CompletedWorksLibrary) │ │ │ └── Stage2PaperHistory.css # Tier 2 paper history styles @@ -323,8 +329,8 @@ project-root/ │ │ │ ├── LeanOJLogs.jsx # LeanOJ API-call log wrapper; live event stream and proof fragments are shown in the interface/proof tabs │ │ │ ├── LeanOJMasterProof.jsx # Master proof draft tab (on-demand draft, metadata, edit history, download) │ │ │ ├── LeanOJMasterProof.css # Master proof draft tab styles -│ │ │ ├── LeanOJMathematicalProofs.jsx # Current-run verified LeanOJ proof/proof-fragment viewer -│ │ │ ├── LeanOJProofLibrary.jsx # Cross-session completed LeanOJ proof-work library +│ │ │ ├── LeanOJMathematicalProofs.jsx # Flat current-run verified proof/proof-fragment viewer; suppresses run-prompt fallback fields without truncating proof content +│ │ │ ├── LeanOJProofLibrary.jsx # Historical completed-run library with collapsed prompt groups and bounded prompt-header previews; excludes the currently loaded run │ │ │ └── index.js # LeanOJ component exports │ │ │ │ │ ├── StartupProviderSetupModal.jsx # Post-disclaimer startup chooser for OpenRouter or LM Studio setup; desktop OAuth is shown only as an after-startup add-on because embeddings require OpenRouter/LM Studio (OpenRouter-only in generic mode) @@ -339,8 +345,10 @@ project-root/ │ │ ├── CreditExhaustionNotificationStack.jsx # Persistent red notifications for OpenRouter credit exhaustion with "Retry OpenRouter" reset button │ │ ├── BoostControlModal.jsx # Modal for boost configuration (next-X, category, always-prefer) with mode-aware copy │ │ ├── BoostControlModal.css # Boost control modal styles -│ │ ├── WorkflowPanel.jsx # Boost controls panel (Boost Next X, Always Prefer, Category Boost, token stats, research timer) -│ │ ├── WorkflowPanel.css # Boost controls panel styles +│ │ ├── WorkflowPanel.jsx # Boost/token/Assistant panel plus activation-gated, stale-fenced Progressive Solution Path card and lifecycle refresh +│ │ ├── WorkflowPanel.css # Boost controls panel and Assistant Memory Bank styles +│ │ ├── SolutionPathModal.jsx # Advisory route viewer with repair guidance, settings routing, retry, focus restoration, and accessible status +│ │ ├── SolutionPathModal.css # Progressive Solution Path modal styles │ │ ├── TextFileUploader.jsx # User file upload component │ │ ├── TextFileUploader.css # File uploader styles │ │ ├── OpenRouterPrivacyWarningModal.jsx # Privacy policy error modal (OpenRouter data sharing, capability-aware alternatives) @@ -368,7 +376,7 @@ project-root/ │ │ │ ├── researchRunHistory.js # Groups Tier 2 papers + final answers into per-run history entries for Stage2PaperHistory/FinalAnswerLibrary │ │ │ └── disclaimerHelper.js # Frontend-only disclaimer injection for brainstorm/paper views │ │ │ -│ │ ├── App.jsx # Main app shell with top-level mode switch, `/api/features` capability bootstrap, capability propagation, and developer-mode raw-settings shortcut +│ │ ├── App.jsx # Main shell, capabilities/mode routing, global solution-path modal/settings ownership, and workflow-scoped path activity dispatch │ │ ├── index.css # Styles │ │ └── index.jsx # React entry point │ │ @@ -444,13 +452,17 @@ project-root/ - `runtime_settings.py`: Persists non-secret process settings such as free-model looping, desktop/default-mode proof runtime flags/timeouts, and connectivity feature toggles under the active data root - `provider_notification_store.py`: Persists recent non-secret provider/OAuth failure notifications so missed live popups can hydrate after reconnect/reload - `build_info.py`: Build identity helper that reads the committed manifest contract, resolves git HEAD or ZIP-stamped build commits, and applies optional env overrides for runtime version/build stamping -- `proof_search/assistant_coordinator.py` / `assistant_ranker.py` / `assistant_cache.py` / `assistant_models.py`: Shared Assistant memory-support infrastructure. Assistant is a configured non-blocking LLM role that selects up to 7 verified proof supports from enabled proof-history corpora, reuses useful packs for two eligible receiver reads before refresh, persists cooldown/shutdown state for repeated unhelpful retrieval, clears stale live packs on reset/clear paths, and disables when Session History Memory is off. +- `proof_identity.py`: Defines `moto-proof-identity-v1` canonical theorem/Lean hashes for internal exact matching; publisher hashes remain separate integrity metadata. +- `proof_search/indexer.py` / `models.py` / source normalizers: Derived SQLite schema v2 with canonical columns, schema+identity compatibility validation, atomic sibling publication, and composite `(corpus, run_id, session_id, source_type, source_id)` occurrence joins. +- `proof_search/assistant_coordinator.py` / `assistant_ranker.py` / `assistant_cache.py` / `assistant_models.py`: Non-blocking up-to-7 verified Lean proof-history support selection with deterministic lanes, two-receiver reuse, and requesting-run/session filtering on fresh and every cached/stale reuse path. Supports are optional and cannot redirect or mathematically reinterpret an unrelated parent goal. +- `solution_path/models.py` / `engine.py` / `integration.py` / `registry.py` / `reviewer.py`: One durable Progressive Solution Path per top-level run, absent before five parent-owned accepted brainstorm events. Only semantic validators may enqueue material optional updates; separately parsed malformed path data cannot affect primary decisions. `reviewer.py` owns the production Main Submitter 1 prompt and bounded sanitized JSON retry. Review is non-blocking, serial, transactional, and generation-fenced; immediate hard failures and repeated invalid/unknown failures use explicit repair/retry. Only the bounded approved plan is advisory context; Stop/crash preserves work and Clear/new-run is the only reset. - `fastembed_provider.py`: FastEmbed embedding wrapper (generic mode only); lazy-imported so default installs are unaffected - `lean4_client.py`: Lean 4 proof checker client. Subprocess mode by default; optional persistent LSP mode when `lean4_lsp_enabled`. When `lean4_enabled=False`, proof checks return explicit disabled/error results rather than invoking Lean. Never bundled into the hosted image. - `smt_client.py`: Optional Z3/SMT launcher-managed subprocess wrapper. When `smt_enabled=False`, SMT checks return explicit disabled/error results rather than invoking Z3. SMT results are hint-only; Lean 4 remains authoritative. Never bundled into the hosted image. ### Compiler Components +- Ordinary outline, construction, review, post-body critique, and semantic validation are domain-general and exact-objective-directed, with rigor matched to the domain and claim type. Mathematics, theorems, proofs, and LaTeX are first-class when relevant; dedicated theorem discovery/formalization, Lean placement/checks, markers, and Wolfram paths retain their specialized behavior. - `compiler_coordinator.py`: Orchestrates Markov chain workflow, mode switching - `compiler_rag_manager.py`: Per-role context windows, direct→RAG priority - `outline_memory.py`, `paper_memory.py`, `compiler_rejection_log.py`: File I/O and logging @@ -462,17 +474,18 @@ project-root/ ### Autonomous Research Components +- Ordinary strategy prompts (`topic_exploration`, topic selection, completion, references, titles, and continuation) are domain-general, aggressively solution-directed, and use domain-/claim-appropriate rigor. Tier 3 certainty/format/volume synthesis follows the same standard: it preserves evidence status, organizes by actual solution dependencies, and permits proof, mechanism, implementation, evidence, validation, safety, and risk components without changing schemas or control flow. When Mathematical Proofs is allowed, mathematics remains first-class under the eager startup proof-framing emphasis when useful; when disabled in Allowed Outputs, that gateway and emphasis injection are skipped. Positive framing is emphasis rather than a hard mathematical admission gate for ordinary contributions. - `autonomous_coordinator.py`: Three-tier workflow orchestrator (Tier 1→2→3, triggers, crash recovery, invokes `ProofVerificationStage` after brainstorm/Tier 2 paper completion when `lean4_enabled`) - `autonomous_rag_manager.py`: Autonomous RAG wrapper - `proof_verification_stage.py`: Proof pipeline orchestrator — prompt-relevant candidate identification with bounded source-title/brainstorm-topic metadata → per-candidate Phase A (Mathlib lemma search → optional SMT early-exit → Lean 4 formalization attempts, currently 3 full-script plus 2 tactic-script attempts per candidate) runs across all identified candidates with `proof_max_parallel_candidates` batching (default `6`; `0` = unlimited; positive values = strict batch size) → Phase B (novelty check → `add_proof` → `ProofDependency` extraction → brainstorm/paper `append_proofs_section`) remains strictly serialized in Phase-A completion order. Per-source reservation lock prevents duplicate concurrent checks for the same `{source_type}:{source_id}`; account-credit exhaustion cancels sibling tasks and preserves checkpointed progress for provider pause/retry at the coordinator boundary. Compiler rigor mode remains serial, separate from this parallel proof pipeline, and capped at 5 consecutive rigor cycles. - `proof_novelty.py`: Shared novelty classifier for Lean-verified proofs, used by autonomous proof verification and compiler rigor persistence. -- `proof_registration.py`: Shared verified-proof registration helper used by autonomous, compiler, aggregator, and LeanOJ proof flows. +- `proof_registration.py`: Shared verified-proof registration that preserves independent public/current-context novelty, stores every current occurrence, then adds silent cross-run canonical duplicate provenance/overlay across enabled history. - `proof_dependency_extractor.py`: Parses verified Lean 4 code into `ProofDependency` records (imports, Mathlib lemmas, MOTO-origin proof ancestry). - Agents: `topic_selector.py`, `topic_validator.py`, `completion_reviewer.py`, `reference_selector.py`, `paper_title_selector.py`, `proof_identification_agent.py`, `proof_formalization_agent.py`, `lemma_search_agent.py` - Tier 3 Agents: `certainty_assessor.py`, `answer_format_selector.py`, `volume_organizer.py` - `paper_redundancy_checker.py`: Library quality maintenance (every 3 papers) - Prompts: `topic_prompts.py`, `topic_exploration_prompts.py`, `completion_prompts.py`, `paper_reference_prompts.py`, `paper_title_exploration_prompts.py`, `paper_title_prompts.py`, `paper_redundancy_prompts.py`, `paper_continuation_prompts.py`, `proof_prompts.py`, `final_answer_prompts.py` -- Memory: `brainstorm_memory.py`, `paper_library.py`, `research_metadata.py` (also stores the proof runtime config snapshot), `session_manager.py`, `autonomous_rejection_logs.py`, `topic_exploration_memory.py` (in-memory candidate DB), `paper_model_tracker.py` (per-paper model usage tracking and author attribution), `autonomous_api_logger.py` (API call logging singleton), `proof_database.py` (session-aware Lean 4 proof storage + novelty index + reverse Mathlib index + cross-session library access), `final_answer_memory.py` (model tracking, archival) +- Memory: `brainstorm_memory.py`, `paper_library.py`, `research_metadata.py` (also stores the proof runtime config snapshot), `session_manager.py`, `autonomous_rejection_logs.py`, `topic_exploration_memory.py` (in-memory candidate DB), `paper_model_tracker.py` (per-paper model usage tracking and author attribution), `autonomous_api_logger.py` (API call logging singleton), `proof_database.py` (session-aware occurrence-preserving Lean proof storage, prompt-novel injection filtering, novelty/reverse-Mathlib indexes, and cross-session access), `final_answer_memory.py` (model tracking, archival) ### LeanOJ Components @@ -485,17 +498,17 @@ project-root/ - `compiler.py`: Compiler control (start/stop/status), paper/outline access, critique management - `autonomous.py`: Autonomous research control (start/stop/clear/status), brainstorm/paper access, pruned/history paper routes, Tier 3/final-answer library routes, critique/API-log helpers, and current-paper recovery actions - `proofs.py`: Proof database listing (`GET /`, `/novel`, `/known`) and `/status` runtime readiness — always available, never gated. Current proof listing/detail/certificate/dependency/graph routes accept optional `scope=autonomous|manual`; manual scope reads the active manual writer proof store under the active data root rather than the autonomous session store. `/{id}/certificate` and `/{id}/certificate.lean` — always available (data is stored on disk; Lean version info populated only when Lean is enabled). `/status` uses `asyncio.wait_for` timeouts (5s Lean, 3s Z3) so the endpoint never hangs. `POST /settings` runtime flag updates. `POST /check` manual proof check, `/{id}/dependencies`, `/graph`, `/mathlib/{lemma}/dependents` graph/lineage queries — gated on `lean4_enabled`. `GET /library` + `GET /library/{session_id}/{proof_id}` cross-session proof library endpoints accept `scope=autonomous|manual`; manual library scope reads archived manual proof runs only and never active prompt context. -- `syntheticlib4.py`: SyntheticLib4 corpus access/status routes for local snapshot status, release listing, safe local snapshot import, refresh/reindex, retrieve-batch, account-proof browsing/search, and production OAuth placeholders. API keys use the active mode's secret path while live validation remains pending; local mock/offline search uses data-root snapshots first, then test fixtures when present, then built-in fallback records when packaged fixtures are unavailable. +- `syntheticlib4.py`: SyntheticLib4 corpus access/status routes for local snapshot status, release listing, safe local snapshot import, refresh/reindex, retrieve-batch, account-proof browsing/search, and production OAuth placeholders. API keys use the active mode's secret path while live validation remains pending; normal runtime reports zero SyntheticLib4 proofs until an authorized data-root snapshot is active, and explicit fixtures are test-only. - `connectivity.py`: Non-secret grouped status and toggle routes (`/api/connectivity/status`, `/api/connectivity/toggles`) for OpenRouter/OAuth, LM Studio, SyntheticLib4, local proof-history Session History Memory, Wolfram Alpha, and Boost state. Toggle changes never clear credentials, snapshots, indexes, or history. -- `leanoj.py`: LeanOJ proof-solver routes for start (including matching saved-progress resume), stop, status, clear, skip-brainstorm, force-brainstorm, current proof listing via `/api/leanoj/proofs`, cross-session library via `/api/leanoj/library*`, plus read-only `GET /api/leanoj/master-proof` and `/api/leanoj/master-proof/edits` for the durable master proof draft and compact edit-history summaries. +- `leanoj.py`: LeanOJ proof-solver routes for start (including matching saved-progress resume), stop, status, clear, skip-brainstorm, force-brainstorm, current proof listing via `/api/leanoj/proofs`, historical completed-run library via `/api/leanoj/library*` (excluding the currently loaded run), plus read-only `GET /api/leanoj/master-proof` and `/api/leanoj/master-proof/edits` for the durable master proof draft and compact edit-history summaries. ### Frontend Components -- `App.jsx`: Top-level GUI shell. Default mode is `Autonomous S.T.E.M. ASI` for Part 3 screens; `Advanced Manual S.T.E.M. ASI` contains the manual Part 1 Aggregator + Part 2 Compiler workspace; `LeanOJ Proof Solver` is a developer-mode-only proof mode. Shared utility controls (ConnectivityPanel, WorkflowPanel, and BoostControlModal from the panel) remain global, and Build 3C bootstraps `/api/features` here so hosted mode can hide LM Studio/desktop-OAuth UI and copy. Shift + Z + X toggles persisted developer-mode settings, LeanOJ mode, raw JSON editors, and Supercharge controls. Supercharge request payloads must be forced off unless developer mode is active. Active app mode and tab state are in-memory only; a fresh frontend mount starts on the autonomous main interface. **Autonomous tab groups**: main tabs (interface, brainstorms, papers, proofs, optional final-answer) + settings group (Your Completed Works Library, API Call Logs, Settings). The "Your Completed Works Library" tab hosts three sub-tabs rendered inside its content area: Stage 2 Papers History, Stage 3 Final Answers History, and Proof Library. +- `App.jsx`: Top-level GUI shell. Default mode is `Autonomous S.T.E.M. ASI` for Part 3 screens; `Advanced Manual S.T.E.M. ASI` contains the manual Part 1 Aggregator + Part 2 Compiler workspace; `LeanOJ Proof Solver` is a developer-mode-only proof mode. Shared utility controls (ConnectivityPanel, WorkflowPanel, and BoostControlModal from the panel) remain global, and Build 3C bootstraps `/api/features` here so hosted mode can hide LM Studio/desktop-OAuth UI and copy. Shift + Z + X toggles persisted developer-mode settings, LeanOJ mode, raw JSON editors, and Supercharge controls. Supercharge request payloads must be forced off unless developer mode is active. Active app mode and tab state are in-memory only; a fresh frontend mount starts on the autonomous main interface. **Autonomous tab groups**: main tabs (interface, brainstorms, papers, proofs, optional final-answer) + settings group (Completed Works Library, API Call Logs, Settings). The "Completed Works Library" tab hosts three sub-tabs rendered inside its content area: Stage 2 Papers History, Stage 3 Final Answers History, and Proof Library. - Live activity feeds keep long bounded histories (thousands of entries) and persist in browser storage across tab reloads, backend restarts, stop/start cycles, and crashes; explicit mode clear/reset actions are the user-facing reset point. - **Aggregator**: `AggregatorInterface.jsx`, `AggregatorSettings.jsx`, `AggregatorLogs.jsx`, `LiveResults.jsx` - **Compiler**: `CompilerInterface.jsx`, `CompilerSettings.jsx`, `CompilerLogs.jsx`, `LivePaper.jsx` -- **Autonomous**: `AutonomousResearchInterface.jsx`, `BrainstormList.jsx`, `PaperLibrary.jsx`, `AutonomousResearchSettings.jsx`, `AutonomousResearchLogs.jsx`, `LivePaperProgress.jsx`, `LiveTier3Progress.jsx`, `FinalAnswerView.jsx`, `FinalAnswerLibrary.jsx` (Stage 3 history sub-tab), `ArchiveViewerModal.jsx`, `MathematicalProofs.jsx` (scoped live proof/status/manual-check/certificate tab used by Autonomous and Manual modes), `ProofGraph.jsx` (dependency graph), `ProofNotificationStack.jsx` (novel-proof popups), `ProofLibrary.jsx` (cross-session proof library sub-tab), `Stage2PaperHistory.jsx` (Stage 2 history sub-tab) +- **Autonomous/Manual proofs**: `MathematicalProofs.jsx` is the flat current-run view for both scopes and never groups by/displays the user prompt. `ProofLibrary.jsx` is historical-only, groups by run/prompt with bounded prompt previews, and keeps expanded proof material complete; Autonomous hosts it in Completed Works and Manual mode uses a separate Completed Proof Works tab. - **LeanOJ**: `LeanOJInterface.jsx`, `LeanOJBrainstorms.jsx`, `LeanOJLogs.jsx`, `LeanOJMasterProof.jsx`, `LeanOJMathematicalProofs.jsx`, `LeanOJProofLibrary.jsx`, `LeanOJSettings.jsx` - **Shared**: `StartupProviderSetupModal.jsx`, `ConnectivityPanel.jsx`, `OpenRouterApiKeyModal.jsx`, `SyntheticLib4AccessModal.jsx`, `WolframAlphaAccessModal.jsx`, `LMStudioConnectivityModal.jsx`, `AgentConversationMemoryModal.jsx`, `PaperCritiqueModal.jsx`, `CritiqueNotificationStack.jsx`, `CreditExhaustionNotificationStack.jsx`, `BoostControlModal.jsx`, `WorkflowPanel.jsx`, `TextFileUploader.jsx`, `OpenRouterPrivacyWarningModal.jsx`, `UpdateNotificationBanner.jsx`, `LatexRenderer.jsx` (dual view, KaTeX, theorem parsing), `LatexRenderer.css` - **Hooks**: `useProofCheckRuntime.js` (reads `/api/proofs/status` + runtime config so UI can enable/disable manual proof-check controls) diff --git a/.cursor/rules/rag-design-for-overall-program.mdc b/.cursor/rules/rag-design-for-overall-program.mdc index 8158900..2f7b8d0 100644 --- a/.cursor/rules/rag-design-for-overall-program.mdc +++ b/.cursor/rules/rag-design-for-overall-program.mdc @@ -14,6 +14,8 @@ Some inputs are **mandatory direct-inject** and must never be RAG'd, summarized, If an item is direct injected, its RAG counterpart must NOT also be included. +The active Progressive Solution Path is never RAG-indexed. Before five accepted brainstorm ideas it is absent; afterward, only the bounded Main Submitter 1-approved canonical revision is optional advisory direct context for relevant solving and semantic-validator calls. It may be shed before mandatory user/source/candidate/tool context when budget requires, and pending, rejected, stale, and historical revisions never enter model context. The path is not evidence and may always be ignored for a better route. + ### Paper-Writing / Research Offload Order These priorities apply to Aggregator, Compiler, and Autonomous paper-writing workflows. They do **not** describe LeanOJ proof-only memory ordering. @@ -97,6 +99,9 @@ User-uploaded files: pre-generate ALL 4 configurations. Dynamic files (training - **Global RAG Lock**: ChromaDB write operations acquire lock to prevent race conditions. Embedding API calls also acquire lock in default mode. - **Generic mode lock skip**: FastEmbed is in-process and thread-safe — embedding calls skip the global RAG lock. ChromaDB write locking remains in both modes. +- **Chroma derived-state lifecycle**: `chroma_db` is derived state. Windows desktop atomically replaces prior-process Chroma state once per runtime root before the first native open and lazily re-indexes durable sources; hosted Linux retains persistent-cache behavior. Interrupted quarantines are recovered without inspecting suspect collections in the application process. +- **Transactional native boundary**: Resolve collection handles only after lifecycle initialization while holding the global RAG lock. Native lookup/write/delete sequences and corresponding in-memory chunk/LRU/BM25 commits share one owner-safe transaction; cancellation drains the worker before releasing ownership. +- **Single data-root owner**: One backend process holds the authoritative OS-backed lease for each active data root for its full lifespan. Launcher metadata is advisory and cannot replace this lock. - **Read retry**: Vector search auto-retries with exponential backoff (0.5s → 1s → 2s, max 3 attempts) on HNSW index errors during concurrent writes - **Embedding rate limiting**: Semaphore limits concurrent embedding requests to 2 (default mode only; generic mode uses in-process FastEmbed) - **FastAPI event-loop safety**: `rag_manager` must not run synchronous ChromaDB calls or CPU-heavy scoring directly on the event loop. Use `asyncio.to_thread()` for ChromaDB `add/query/get/delete` and for large in-memory vector/BM25 scoring. @@ -209,7 +214,7 @@ When the content item is the SyntheticLib4 / MOTO unified proof-search result bl **Autonomous (Part 3)**: Per-topic brainstorm databases; paper-compilation references and prior brainstorm papers are loaded as high-priority RAG evidence, while the current brainstorm DB is direct source context for construction/retroactive correction and paper-writing rigor/proof mode when available. Metadata and browsing agents use bounded summaries/abstracts when appropriate; all agents validate prompt size before LLM calls. -**Proof Verification Stage (optional, gated on `lean4_enabled`)**: Proof identification, formalization, and lemma search agents operate outside the RAG pipeline and use mandatory direct source context rather than excerpt-only RAG. Candidate discovery is impact-first and skips not-novel/missing-tier candidates before Lean cost. Verified `ProofRecord` summaries and `FailedProofCandidate` hints (from `proof_prompts.format_failure_hints_for_injection`) are **highest-priority direct injections** into subsequent brainstorm/paper submitter prompts when present — never RAG'd. Compiler rigor/paper-writing proof mode direct-injects available source brainstorm/aggregator context alongside the current paper; supplemental references/prior papers remain RAG evidence. SyntheticLib4 / MOTO proof-search result blocks are optional proof examples capped at 7 combined results; they are rag-able, but only after every other rag-able optional source has been offloaded. Lean source files under the session `proofs/` directory are not indexed into Chroma. +**Proof Verification Stage (optional, gated on `lean4_enabled`)**: Proof identification, formalization, and lemma search use mandatory direct source context rather than excerpt-only RAG. Candidate discovery is impact-first. Only independently prompt-novel, non-`duplicate_novel` `ProofRecord` summaries and unresolved active-run `FailedProofCandidate` hints are highest-priority direct injections; `duplicate_novel` and `not_novel` remain searchable/visible but do not enter this block. Compiler proof mode direct-injects active source context; references remain RAG evidence. Optional SyntheticLib4/MOTO search examples cap at 7 and are last-offloaded optional context. Session Lean source files are not indexed into Chroma. **LeanOJ Proof Solver**: LeanOJ useful proof memory uses the existing RAG pipeline through `backend/leanoj/core/leanoj_context.py`, not a separate/simple retriever. Mandatory prompt inputs (user problem, Lean template, role task, JSON schema) stay direct. Useful artifacts are persisted in full and indexed under session-scoped `leanoj_{session_id}_*` sources when they are eligible for that phase. Final proof editing receives verified subproofs plus accepted `active_plan` notes as proof evidence; ordinary partial scaffolds and failed attempts are excluded except as immediate compact execution feedback. Current final-cycle failure packets are direct context for the next brainstorm/proof-fragment phase; older final-cycle packets remain available through scoped RAG only. Recent rejection/error summaries remain capped direct feedback. During final proof-editing, allocation is narrower: no historical final-cycle packets, no failed-attempt counts, and no phase-transition/path vocabulary; the prompt may still include the most recent 5 final attempts as capped execution feedback so the solver does not repeat stale edits or ignored Lean errors. Validator feedback from rejected non-progressive master-proof shortening edits may be direct feedback because it tells the next final solver what proof progress to restore. The canonical LeanOJ master proof draft (`master_proof.lean`) is file-backed working state, not a RAG artifact: during the final proof-editing loop it is mandatory direct-inject context and must be shown fully or the program must halt with a mandatory direct-context overflow error. Edits always apply to the full persisted proof. diff --git a/.cursor/rules/workflow-cross-field-testing.mdc b/.cursor/rules/workflow-cross-field-testing.mdc new file mode 100644 index 0000000..ba7f80f --- /dev/null +++ b/.cursor/rules/workflow-cross-field-testing.mdc @@ -0,0 +1,39 @@ +--- +description: MOTO cross-field workflow testing and deep verification requirements +alwaysApply: true +--- + +# Cross-Field Workflow Testing + +## #1 Rule: The Testing System Must Remain a Removable Overlay + +The testing system is its own system that sits on top of the current production +program. Production code, launch paths, builds, and runtime behavior must never depend +on the testing folders, test artifacts, test-only imports, or test-only state. A user +must be able to delete the entire testing system (including `tests/` and generated test +artifacts) and still have a fully functioning MOTO program with unchanged production +behavior. This separability requirement takes priority over every other testing rule. + +MOTO uses a project-specific testing overlay that combines executable product-law +invariants, deterministic state recombination, bounded real-code adapters with faked +dependencies, isolated scenario/result artifacts, inverse target-risk analysis, and a +support graph. + +- Treat `tests/workflow_harness/invariant_catalog.py` and + `tests/workflow_testing_plans/02_invariant_catalog.md` as the stable product-law catalog. + Add invariant IDs instead of renaming existing IDs. +- Keep the overlay removable and behavior-neutral. Never add production dependencies on + the testing system, runtime shortcuts, hidden stops/caps, impossible state transitions, + or production-only test behavior. +- Keep external providers, Lean/SMT, browser services, credentials, and mutable runtime + roots faked, blocked, or isolated unless a test explicitly owns that integration. +- Record safely unobservable interactions as `blocked`; never report synthetic coverage + as a real-code pass. +- Run focused tests while developing and `npm run test:workflows` after workflow-test or + deterministic-artifact changes. +- Run `npm run test:deep` after every major build and after substantial orchestration, + persistence, provider, proof, prompt/RAG, event-contract, runtime-root, or cross-mode + change. It is additive: run `npm test` separately for normal backend/frontend coverage. +- Regenerate deterministic cross-field artifacts with + `python -m tests.workflow_cross_field.artifacts` when their inputs change, then verify + that the checked-in projection is clean. diff --git a/.github/codeql/codeql-config.yml b/.github/codeql/codeql-config.yml new file mode 100644 index 0000000..0afacf8 --- /dev/null +++ b/.github/codeql/codeql-config.yml @@ -0,0 +1,6 @@ +name: MOTO CodeQL configuration + +paths-ignore: + - backend/data/** + - frontend/dist/** + - frontend/node_modules/** diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c4636e7..b63c842 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -36,6 +36,66 @@ jobs: - name: Run Python tests run: python -m pytest + deep-workflow-tests: + name: Deep workflow containment + runs-on: ubuntu-latest + timeout-minutes: 45 + steps: + - name: Check out repository + uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: "3.11" + cache: pip + + - name: Install Python dependencies + run: | + python -m pip install --upgrade pip + python -m pip install -r requirements.txt + python -m pip install pytest pytest-asyncio + + - name: Run deep workflow suite + run: python tests/run_deep_workflows.py 2>&1 | tee deep-workflow.log + + - name: Upload deep workflow failure log + if: failure() + uses: actions/upload-artifact@v4 + with: + name: deep-workflow-failure-log + path: deep-workflow.log + if-no-files-found: error + retention-days: 7 + + windows-native-lifecycle: + name: Windows native Chroma lifecycle + runs-on: windows-latest + timeout-minutes: 15 + steps: + - name: Check out repository + uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: "3.11" + cache: pip + + - name: Install Python dependencies + run: | + python -m pip install --upgrade pip + python -m pip install -r requirements.txt + python -m pip install pytest pytest-asyncio + + - name: Run native and lifecycle race regressions + run: >- + python -m pytest + tests/regressions/test_chroma_native_lifecycle.py + tests/regressions/test_rag_clear_bounded.py + tests/unit/test_chroma_cache_cleanup.py + tests/test_workflow_start_guard_ownership.py + frontend-build: name: Frontend build runs-on: ubuntu-latest @@ -59,5 +119,25 @@ jobs: - name: Audit production frontend dependencies run: npm audit --omit=dev --audit-level=high + - name: Run frontend tests + run: npm run test + - name: Build frontend run: npm run build + + - name: Install Playwright Chromium + run: npx playwright install --with-deps chromium + + - name: Run browser smoke tests + run: npm run test:browser + + - name: Upload browser smoke artifacts + if: failure() + uses: actions/upload-artifact@v4 + with: + name: browser-smoke-artifacts + path: | + playwright-report/ + test-results/playwright/ + if-no-files-found: ignore + retention-days: 7 diff --git a/.gitignore b/.gitignore index 01c6f69..3909e16 100644 --- a/.gitignore +++ b/.gitignore @@ -96,8 +96,10 @@ backend/data/auto_api_log.txt backend/data/aggregator_results.txt backend/data/manual_aggregator_prompt.txt backend/data/manual_compiler_prompt.txt +backend/data/manual_main_submitter_config.json backend/data/runtime_settings.json backend/data/provider_notifications.json +backend/data/.moto_backend.lock backend/data/chroma_db/* !backend/data/chroma_db/.gitkeep @@ -106,6 +108,7 @@ backend/data/boost_api_log.txt backend/data/boost_state.json backend/data/model_cache.json backend/data/proof_search/ +backend/data/solution_paths/ backend/data/syntheticlib4/ backend/data/paper_version_*.txt backend/data/critique_feedback_*.txt @@ -125,6 +128,8 @@ backend/logs/* .pytest_cache/ .coverage htmlcov/ +playwright-report/ +test-results/playwright/ # Temporary files *.tmp diff --git a/Dockerfile b/Dockerfile index a07ae6a..fb80599 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,3 +1,14 @@ +# MOTO HOSTED API-ONLY IMAGE +# +# This image is intended for deployment behind MOTO's authenticated +# control-plane proxy, for web host deployment. It is not currently a standalone desktop/web image that allows GUI access. This system is in development and contains bugs. +# +# Required at runtime: +# MOTO_INSTANCE_ID +# MOTO_INTERNAL_PROXY_SECRET +# +# For ordinary local use, run the Windows or Ubuntu desktop launcher, i.e. "Click to Launch MOTO.bat" for Windows. + FROM python:3.12-slim ENV PYTHONDONTWRITEBYTECODE=1 \ diff --git a/README.md b/README.md index b40f325..48623a8 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,20 @@ # MOTO Autonomous ASI -## Autonomous Prototype Superintelligence - Automated Theorem Generation with Lean 4 Math Proof Verification -**Version: 1.1.02** +## Autonomous Prototype Superintelligence for Rigorous Research and Solution Generation, with Optional Lean 4 Proof Verification +**Version: 1.1.03** [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/) [![Node.js 20.19+](https://img.shields.io/badge/node-20.19+-green.svg)](https://nodejs.org/) +[![ChatGPT Subscription](https://img.shields.io/badge/ChatGPT_Subscription-Codex_OAuth-10a37f.svg)](https://chatgpt.com/) +[![SuperGrok Subscription](https://img.shields.io/badge/SuperGrok_Subscription-xAI_OAuth-black.svg)](https://grok.com/) +[![Sakana Fugu Subscription](https://img.shields.io/badge/Sakana_Fugu-Subscription_API-e85d75.svg)](https://sakana.ai/fugu/) +[![OpenRouter: API Key](https://img.shields.io/badge/OpenRouter-API_Key-6467f2.svg)](https://openrouter.ai/) +[![LM Studio: Local Models](https://img.shields.io/badge/LM_Studio-Local_Models-7c3aed.svg)](https://lmstudio.ai/) -**A breakthrough in AI automated theorem generation. MOTO is an autonomous research system powered by Intrafere Research Group's new prototype-superintelligence discovery of [Top-P Exploration Through Structured Brainstorming & Validated Feedback](https://intrafere.com/structured-brainstorming-validated-feedback/): a combination of reiterative brainstorming, validation, feedback, and pruning that creates prototype-level superintelligence using creative/combinatory multi-model data from nearly any combination of AI models. When enabled, MOTO pairs this exploration with Lean 4 machine-checked proofs for the exact formal theorem statements it successfully proves.** -MOTO generates novel and publication-worthy research papers, and it can formalize candidate theorems and lemmas in Lean 4 while only storing proofs that Lean 4 accepts as mathematically verified. Lean 4 automation gives the user machine-checked verification for the exact formal statements produced, while informal papers and interpretations should still be reviewed with scrutiny. Unlike programs that may look similar, MOTO finds novel solutions with 0 creative user input required after launch: press start and let it run for hours or days while it autonomously searches for new solution paths. This exact version of MOTO is customized to be useful for any discipline with an interest in creative and novel solution generation in S.T.E.M.: physicists, engineers, mathematicians, chemists, researchers, etc. This harness can also easily be modified for topics such as general academic research, chatbots, niche research, robotics, or anything requiring creative output and/or general autonomy. MOTO's novel brainstorming and rejection/validation stage allows autonomous long-term runtime without user intervention — if desired, research can be conducted for days or weeks without user input. +**A breakthrough in AI-automated theorem generation. Press the one-click launcher file "Click To Launch MOTO.bat" to run the program. MOTO is an autonomous research and solution-generation system powered by Intrafere Research Group's prototype-superintelligence discovery of [Top-P Exploration Through Structured Brainstorming & Validated Feedback](https://intrafere.com/structured-brainstorming-validated-feedback/): a combination of reiterative brainstorming, validation, feedback, and pruning that creates prototype-level superintelligence using creative/combinatory multi-model data from nearly any combination of AI models. When relevant and enabled, MOTO pairs this exploration with automated theorem generation and Lean 4 machine-checked proofs for the exact formal statements it successfully proves.** + +MOTO is a fully automated invention and research generator. It generates novel and publication-worthy research papers, and it can formalize candidate theorems and lemmas in Lean 4 while only storing proofs that Lean 4 accepts as mathematically verified. Lean 4 automation gives the user machine-checked verification for the exact formal statements produced, while informal papers and interpretations should still be reviewed with scrutiny. Unlike programs that may look similar, MOTO finds novel solutions with 0 creative user input required after launch: press start and let it run for hours or days while it autonomously searches for new solution paths. This exact version of MOTO is customized to be useful for any discipline with an interest in creative and novel solution generation in S.T.E.M.: physicists, engineers, mathematicians, chemists, researchers, etc. This harness and the discovery of Top-P exploration can also easily be modified for topics such as chatbots, niche research, AI CAD and 3D model generation, AI image generation, robotics, or anything requiring creative output and/or general autonomy. MOTO's novel brainstorming and rejection/validation stage allows autonomous long-term runtime without user intervention — if desired, research can be conducted for days or weeks without user input. ### The Core Discovery: Top-P Exploration @@ -38,9 +44,11 @@ Give the program a try — MOTO is as cool as it sounds. Windows has a one-click --- -## Outline of "MOTO - S.T.E.M. Mathematics Variant" +## Outline of MOTO's S.T.E.M. Research and Solution System + +MOTO (Multi-Output Token Orchestrator) is a high-risk high-reward, novelty-seeking S.T.E.M. research and solution system designed to run for days at a time after you press start, without user interaction. Mathematics, theorem discovery, and formal proof remain first-class capabilities whenever relevant. This program can support multiple simultaneous models working in parallel from local LM Studio, OpenRouter API keys, desktop OAuth providers such as OpenAI Codex/ChatGPT and xAI Grok/SuperGrok, or a mix of those providers. -MOTO (Multi-Output Token Orchestrator) is a high-risk high-reward (novelty seeking AI) mathematics researcher designed to run for days at a time after you press start, without user interaction. This program can support multiple simultaneous models working in parallel from local LM Studio, OpenRouter API keys, desktop OAuth providers such as OpenAI Codex/ChatGPT and xAI Grok/SuperGrok, or a mix of those providers. +On Windows desktop, MOTO prevents idle-triggered system sleep while a top-level workflow is active and restores normal power behavior when it stops. The display may still turn off, and explicit Sleep, Hibernate, shutdown, and critical-battery actions are not blocked. Hosted/generic mode does not use this desktop power request. ### Key Features @@ -427,9 +435,11 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file --- -## ⚠️ Disclaimer +## ⚠️ Limitations and Responsible Use -All content generated by this system is for informational purposes only. Papers are autonomously generated with the novelty-seeking MOTO harness without peer review or user oversight beyond the original prompt. AI-generated content may contain fabricated or unverified claims presented with high confidence - all content should be viewed with extreme scrutiny and independently verified before use. Users are responsible for how they use generated content. All users must follow terms of service, conditions, etc. from all 3rd party applications. +MOTO produces autonomous, AI-generated research outputs that may contain incorrect, incomplete, misleading, or fabricated claims. Outputs are not peer reviewed and should not be treated as authoritative or relied upon without independent verification by a qualified reviewer. Users are responsible for evaluating outputs and for complying with the terms and policies of any third-party models, services, or data sources they use with MOTO. + +The software itself is provided under the MIT License. See [LICENSE](LICENSE) for its warranty and liability terms. --- @@ -477,8 +487,10 @@ Because the heavy model inference happens in the cloud, MOTO can run on very mod --- -**Built for autonomous mathematical research in STEM. Powered by multi-agent AI.** +**Built for autonomous rigorous S.T.E.M. research and solution generation, with mathematics and Lean-verified proofs as first-class capabilities. Powered by multi-agent AI.** --- Intrafere™ and Intrafere Research Group™ are trademarks of Intrafere LLC. All rights reserved. + +Third-party names and trademarks belong to their respective owners. Compatibility does not imply endorsement, sponsorship, or partnership. diff --git a/backend/aggregator/agents/submitter.py b/backend/aggregator/agents/submitter.py index 304908a..aba9714 100644 --- a/backend/aggregator/agents/submitter.py +++ b/backend/aggregator/agents/submitter.py @@ -18,16 +18,31 @@ from backend.shared.json_parser import parse_json, sanitize_model_output_for_retry_context from backend.shared.response_extraction import extract_message_text from backend.aggregator.core.context_allocator import ContextAllocationError, context_allocator +from backend.shared.provider_errors import ProviderContextLengthError from backend.aggregator.core.queue_manager import queue_manager from backend.aggregator.memory.shared_training import shared_training_memory from backend.aggregator.memory.local_training import LocalTrainingMemory from backend.aggregator.prompts.submitter_prompts import ( CREATIVITY_EMPHASIS_BOOST_PROMPT, build_submitter_prompt, + get_submitter_json_schema, ) logger = logging.getLogger(__name__) + +def _requires_lean_retry_schema(llm_output: str, lean4_enabled: bool) -> bool: + """Keep malformed Lean candidate retries on the proof-gated response variant.""" + return bool( + lean4_enabled + and ( + '"lean_proof"' in llm_output + or '"lean_code"' in llm_output + or '"theorem_statement"' in llm_output + ) + ) + + class SubmitterAgent: """ Submitter agent that generates submissions. @@ -51,6 +66,7 @@ def __init__( local_rejection_log_template: Optional[str] = None, reset_local_rejection_log_on_initialize: bool = False, assistant_workflow_mode_override: Optional[str] = None, + solution_path_manager: Optional[Any] = None, ): self.submitter_id = submitter_id self.model_name = model_name @@ -64,6 +80,7 @@ def __init__( self.coordinator = coordinator self.creativity_emphasis_boost_enabled = creativity_emphasis_boost_enabled self.assistant_workflow_mode_override = assistant_workflow_mode_override + self.solution_path_manager = solution_path_manager # Per-submitter context settings (fall back to global config if not provided) self.context_window = context_window if context_window is not None else rag_config.submitter_context_window @@ -271,8 +288,12 @@ async def _generate_submission(self) -> Optional[Submission]: # CRITICAL: Verify actual prompt size fits in context window from backend.shared.utils import count_tokens - actual_prompt_tokens = count_tokens(prompt) max_allowed_tokens = rag_config.get_available_input_tokens(self.context_window, self.max_output_tokens) + from backend.shared.solution_path.integration import with_budgeted_solver_plan + prompt = with_budgeted_solver_plan( + prompt, self.solution_path_manager, max_allowed_tokens + ) + actual_prompt_tokens = count_tokens(prompt) if creativity_emphasized and actual_prompt_tokens > max_allowed_tokens: logger.warning( @@ -305,6 +326,9 @@ async def _generate_submission(self) -> Optional[Submission]: creativity_emphasized=False, lean4_enabled=system_config.lean4_enabled, ) + prompt = with_budgeted_solver_plan( + prompt, self.solution_path_manager, max_allowed_tokens + ) actual_prompt_tokens = count_tokens(prompt) if actual_prompt_tokens > max_allowed_tokens: @@ -358,6 +382,23 @@ async def _generate_submission(self) -> Optional[Submission]: call_metadata = api_client_manager.extract_call_metadata(response) break # Success + except ProviderContextLengthError as e: + if self.task_tracking_callback: + self.task_tracking_callback("completed", task_id) + raise ContextAllocationError.from_provider_error( + e, + f"Submitter {self.submitter_id} context overflow or provider context mismatch: " + f"the assembled prompt requires {actual_prompt_tokens:,} tokens and the configured " + f"input budget is {max_allowed_tokens:,} tokens (context window: {self.context_window:,}, " + f"output reserve: {self.max_output_tokens:,}). The provider rejected the request " + "as too large, so the loaded/provider context is smaller than configured. Please condense " + "into a new prompt and restart, select a larger-context model, or reload the local model " + "with the configured context window.", + required_tokens=actual_prompt_tokens, + available_tokens=max_allowed_tokens, + context_window=self.context_window, + output_reserve=self.max_output_tokens, + ) from e except (httpx.HTTPStatusError, ValueError) as e: error_msg = str(e) is_400_or_context = "400" in error_msg or "context" in error_msg.lower() @@ -447,6 +488,23 @@ async def _generate_submission(self) -> Optional[Submission]: # Two-stage conversational retry before final rejection logger.info(f"Submitter {self.submitter_id}: Initial JSON parse failed, attempting conversational retry") logger.debug(f"Parse error: {error}") + + lean_retry_required = _requires_lean_retry_schema( + llm_output, + system_config.lean4_enabled, + ) + retry_schema = get_submitter_json_schema( + lean4_enabled=system_config.lean4_enabled + ) + retry_variant_instruction = ( + "Your failed response is recognizable as a Lean proof candidate. " + "You MUST preserve submission_type=\"lean_proof\" and every required " + "Lean proof field from the schema below; do not convert it into an ordinary idea.\n\n" + if lean_retry_required + else + "Return one complete allowed response variant from the schema below. " + "Do not drop its submission_type or required fields.\n\n" + ) # Stage 1: Guide proper JSON escaping for LaTeX retry_prompt_1 = ( @@ -460,11 +518,9 @@ async def _generate_submission(self) -> Optional[Submission]: "2. Do NOT double-escape: \\\\\\\\mathbb is WRONG, \\\\mathbb is CORRECT\n" "3. Escape quotes inside strings: use \\\" for literal quotes\n" "4. Avoid malformed unicode escapes (must be exactly \\uXXXX with 4 hex digits)\n\n" - "Please provide your submission again in valid JSON format:\n" - "{\n" - ' "submission": "your mathematical submission (LaTeX allowed, escape backslashes)",\n' - ' "reasoning": "your reasoning (LaTeX allowed, escape backslashes)"\n' - "}\n\n" + f"{retry_variant_instruction}" + "Please provide your submission again using this complete schema:\n" + f"{retry_schema}\n\n" "Respond with ONLY the JSON object, no markdown, no explanation." ) @@ -537,11 +593,9 @@ async def _generate_submission(self) -> Optional[Submission]: "- \\mathbb becomes \\\\mathbb in JSON\n" "- \\( becomes \\\\( in JSON\n" "- Do NOT double-escape: \\\\\\\\mathbb is WRONG\n\n" - "Example format:\n" - "{\n" - ' "submission": "For \\\\mathbb{Z}, we have \\\\phi: G \\\\to H",\n' - ' "reasoning": "This establishes the \\\\pi_1 connection"\n' - "}\n\n" + f"{retry_variant_instruction}" + "Use this complete schema and preserve the selected response variant:\n" + f"{retry_schema}\n\n" "Respond with ONLY the JSON, nothing else." ) diff --git a/backend/aggregator/agents/validator.py b/backend/aggregator/agents/validator.py index 6a95575..b8d88dd 100644 --- a/backend/aggregator/agents/validator.py +++ b/backend/aggregator/agents/validator.py @@ -14,7 +14,12 @@ from backend.shared.openrouter_client import FreeModelExhaustedError from backend.shared.json_parser import parse_json, sanitize_model_output_for_retry_context from backend.shared.response_extraction import extract_message_text +from backend.shared.solution_path.integration import ( + enqueue_optional_update, + with_validator_hook, +) from backend.aggregator.core.context_allocator import ContextAllocationError, context_allocator +from backend.shared.provider_errors import ProviderContextLengthError from backend.aggregator.memory.shared_training import shared_training_memory from backend.aggregator.prompts.validator_prompts import ( build_validator_prompt, @@ -51,6 +56,7 @@ def __init__( user_files_content: Dict[str, str], websocket_broadcaster: Optional[Callable] = None, proof_database_store: Optional[Any] = None, + solution_path_manager: Optional[Any] = None, ): self.model_name = model_name self.user_prompt = ( @@ -61,6 +67,7 @@ def __init__( self.user_files_content = user_files_content self.chunk_size = rag_config.validator_chunk_size # Always 512 self.websocket_broadcaster = websocket_broadcaster + self.solution_path_manager = solution_path_manager # Control self.is_running = False @@ -151,6 +158,7 @@ async def _assess_quality(self, submission: Submission) -> ValidationResult: allocation["direct"], rag_evidence ) + prompt = with_validator_hook(prompt, self.solution_path_manager) # CRITICAL: Verify actual prompt size fits in context window from backend.shared.utils import count_tokens @@ -213,6 +221,23 @@ async def _assess_quality(self, submission: Submission) -> ValidationResult: call_metadata = api_client_manager.extract_call_metadata(response) break # Success + except ProviderContextLengthError as e: + if self.task_tracking_callback: + self.task_tracking_callback("completed", task_id) + raise ContextAllocationError.from_provider_error( + e, + "Validator context overflow or provider context mismatch: the assembled prompt " + f"requires {actual_prompt_tokens:,} tokens and the configured validator input budget " + f"is {max_allowed_tokens:,} tokens (context window: {configured_context:,}, " + f"output reserve: {rag_config.validator_max_output_tokens:,}). The provider rejected " + "the request as too large. A complete and honest validation requires direct context " + "injection. Please condense into a new prompt and restart, select a validator model " + "with a larger context window, or reload the local model with the configured context window.", + required_tokens=actual_prompt_tokens, + available_tokens=max_allowed_tokens, + context_window=configured_context, + output_reserve=rag_config.validator_max_output_tokens, + ) from e except (httpx.HTTPStatusError, ValueError) as e: error_msg = str(e) is_400_or_context = "400" in error_msg or "context" in error_msg.lower() @@ -271,6 +296,14 @@ async def _assess_quality(self, submission: Submission) -> ValidationResult: json_valid=False ) + except ProviderContextLengthError as e: + raise ContextAllocationError.from_provider_error( + e, + "Validator context overflow or provider context mismatch during JSON retry: " + "the provider rejected the mandatory validation prompt as too large.", + context_window=context_allocator.validator_context_window, + output_reserve=rag_config.validator_max_output_tokens, + ) from e except RetryableProviderError: raise except Exception as e: @@ -335,6 +368,8 @@ async def _assess_quality(self, submission: Submission) -> ValidationResult: ' "reasoning": "your reasoning here",\n' ' "summary": "brief summary here"\n' "}\n\n" + "Retain the one valid optional top-level `solution_path_update` " + "from your previous response if present; omission remains valid.\n" "CRITICAL: Properly escape all backslashes (use \\\\) and quotes (use \\\").\n" "Respond with ONLY the JSON object, no markdown, no explanation." ) @@ -417,10 +452,17 @@ async def _assess_quality(self, submission: Submission) -> ValidationResult: # Notify task completed successfully if self.task_tracking_callback: self.task_tracking_callback("completed", task_id) - # Extract summary with fallback for rejections # Schema requires summary for rejections, but some models may not provide it decision = parsed["decision"] + await enqueue_optional_update( + parsed, + self.solution_path_manager, + proposer_role=self.role_id, + source_task_id=task_id, + source_phase="submission_validation", + source_decision=decision, + ) summary = parsed.get("summary", "").strip() # For rejections, if summary is missing/empty, use reasoning as fallback @@ -441,6 +483,14 @@ async def _assess_quality(self, submission: Submission) -> ValidationResult: return result + except ProviderContextLengthError as e: + raise ContextAllocationError.from_provider_error( + e, + "Validator context overflow or provider context mismatch: the provider rejected " + "the mandatory validation prompt as too large.", + context_window=context_allocator.validator_context_window, + output_reserve=rag_config.validator_max_output_tokens, + ) from e except ContextAllocationError: raise except Exception as e: @@ -608,6 +658,11 @@ async def _assess_batch_quality(self, submissions: List[Submission]) -> List[Val allocation["direct"], rag_evidence ) + prompt = with_validator_hook( + prompt, self.solution_path_manager, batch=True + ) + # A dual/triple response may originate at most one update for the + # batch as a whole, never one competing update per decision. # Verify prompt size from backend.shared.utils import count_tokens @@ -691,7 +746,7 @@ async def _assess_batch_quality(self, submissions: List[Submission]) -> List[Val ) for s in submissions ] - + # Extract decisions from parsed response decisions_list = parsed.get("decisions", []) @@ -709,7 +764,6 @@ async def _assess_batch_quality(self, submissions: List[Submission]) -> List[Val ) for s in submissions ] - # Verify submission_number fields match expected order for i in range(batch_size): expected_number = i + 1 # 1-indexed @@ -756,6 +810,21 @@ async def _assess_batch_quality(self, submissions: List[Submission]) -> List[Val json_valid=True, metadata={"llm_call": call_metadata} )) + + batch_decisions = {result.decision for result in results} + source_decision = ( + next(iter(batch_decisions)) + if len(batch_decisions) == 1 + else "mixed" + ) + await enqueue_optional_update( + parsed, + self.solution_path_manager, + proposer_role=self.role_id, + source_task_id=task_id, + source_phase="batch_submission_validation", + source_decision=source_decision, + ) # Notify task completed successfully if self.task_tracking_callback: @@ -767,6 +836,16 @@ async def _assess_batch_quality(self, submissions: List[Submission]) -> List[Val return results + except ProviderContextLengthError as e: + raise ContextAllocationError.from_provider_error( + e, + "Validator context overflow or provider context mismatch during batch validation: " + "the provider rejected the mandatory validation prompt as too large.", + required_tokens=locals().get("actual_prompt_tokens"), + available_tokens=locals().get("max_allowed_tokens"), + context_window=context_allocator.validator_context_window, + output_reserve=rag_config.validator_max_output_tokens, + ) from e except (FreeModelExhaustedError, ContextAllocationError, RetryableProviderError): raise except Exception as e: @@ -838,6 +917,9 @@ async def _retry_batch_json_parse( retry_prompt = ( "Your previous response could not be parsed as valid JSON.\n\n" f"Please provide the same validation decisions in valid JSON format:{example_format}\n\n" + "The object may also retain the one valid optional top-level " + "`solution_path_update` from your previous response beside `decisions`; " + "never place it inside an individual decision. Omission remains valid.\n" "CRITICAL: Properly escape all backslashes (use \\\\) and quotes (use \\\").\n" "Respond with ONLY the JSON object, no markdown, no explanation." ) @@ -895,12 +977,64 @@ async def _retry_batch_json_parse( parsed = parse_json(retry_output) logger.info("Batch validator: Conversational retry succeeded!") return parsed, call_metadata + except ProviderContextLengthError as e: + raise ContextAllocationError.from_provider_error( + e, + "Validator context overflow or provider context mismatch during batch JSON retry: " + "the provider rejected the mandatory validation prompt as too large.", + context_window=context_allocator.validator_context_window, + output_reserve=rag_config.validator_max_output_tokens, + ) from e except RetryableProviderError: raise except Exception as e: logger.warning(f"Batch validator: Retry failed - {e}") return None, {} + + async def _retry_validator_json_parse( + self, + original_prompt: str, + failed_output: str, + task_id: str, + ) -> Optional[Dict[str, Any]]: + """Run one bounded repair while preserving an optional path proposal.""" + safe = sanitize_model_output_for_retry_context( + failed_output, max_chars=2000 + ) + repair = ( + "Your previous validator response was invalid JSON. Return the exact " + "same primary decision as one corrected JSON object. Retain the one " + "valid optional top-level `solution_path_update` if present; omission " + "remains valid. Return JSON only." + ) + messages = [ + {"role": "user", "content": original_prompt}, + {"role": "assistant", "content": safe}, + {"role": "user", "content": repair}, + ] + max_input = rag_config.get_available_input_tokens( + context_allocator.validator_context_window, + rag_config.validator_max_output_tokens, + ) + if sum(count_tokens(item["content"]) for item in messages) > max_input: + messages = [ + {"role": "user", "content": original_prompt}, + {"role": "user", "content": repair}, + ] + response = await api_client_manager.generate_completion( + task_id=f"{task_id}_json_retry", + role_id=self.role_id, + model=self.model_name, + messages=messages, + temperature=0.0, + max_tokens=rag_config.validator_max_output_tokens, + ) + if not response.get("choices"): + return None + message = response["choices"][0].get("message", {}) + repaired = parse_json(extract_message_text(message)) + return repaired if isinstance(repaired, dict) else None def _get_system_prompt(self) -> str: """Get system prompt for single submission.""" @@ -1001,6 +1135,7 @@ async def perform_cleanup_review(self) -> Optional[Dict]: context=user_files_context ) logger.info(f"CLEANUP DEBUG: Built cleanup review prompt with direct injection, length: {len(prompt)} chars") + prompt = with_validator_hook(prompt, self.solution_path_manager) # Verify prompt size (should now always fit due to RAG handling) from backend.shared.utils import count_tokens @@ -1039,6 +1174,16 @@ async def perform_cleanup_review(self) -> Optional[Dict]: temperature=0.0, # Deterministic cleanup decisions max_tokens=rag_config.validator_max_output_tokens ) + except ProviderContextLengthError as api_error: + raise ContextAllocationError.from_provider_error( + api_error, + "Validator context overflow or provider context mismatch during cleanup review: " + "the provider rejected the mandatory cleanup prompt as too large.", + required_tokens=actual_prompt_tokens, + available_tokens=max_allowed_tokens, + context_window=context_allocator.validator_context_window, + output_reserve=rag_config.validator_max_output_tokens, + ) from api_error except Exception as api_error: logger.error(f"CLEANUP DEBUG: API call failed: {api_error}") # Notify task completed even on failure @@ -1078,8 +1223,11 @@ async def perform_cleanup_review(self) -> Optional[Dict]: logger.warning(f"CLEANUP DEBUG: JSON PARSE FAILED: {e}") logger.warning(f"CLEANUP DEBUG: Raw output that failed to parse:\n{llm_output}") logger.warning(f"Cleanup review: JSON parse failed: {e}") - return None - + parsed = await self._retry_validator_json_parse( + prompt, llm_output, task_id + ) + if parsed is None: + return None # Check if removal is recommended should_remove = parsed.get("should_remove", False) submission_number = parsed.get("submission_number") @@ -1097,6 +1245,15 @@ async def perform_cleanup_review(self) -> Optional[Dict]: logger.warning("CLEANUP DEBUG: INVALID RESPONSE - should_remove=true but no submission_number provided") logger.warning("Cleanup review: should_remove=true but no submission_number provided") return None + + await enqueue_optional_update( + parsed, + self.solution_path_manager, + proposer_role=self.role_id, + source_task_id=task_id, + source_phase="cleanup_review_validation", + source_decision="accept", + ) logger.info(f"CLEANUP DEBUG: REMOVAL PROPOSED - submission #{submission_number}") logger.info( @@ -1109,7 +1266,7 @@ async def perform_cleanup_review(self) -> Optional[Dict]: "reasoning": reasoning } - except FreeModelExhaustedError: + except (FreeModelExhaustedError, ContextAllocationError): raise except RetryableProviderError: raise @@ -1199,6 +1356,7 @@ async def validate_removal( all_submissions_formatted=all_submissions ) logger.info(f"CLEANUP DEBUG: Built removal validation prompt with direct injection, length: {len(prompt)} chars") + prompt = with_validator_hook(prompt, self.solution_path_manager) # Verify prompt size (should now always fit due to RAG handling) from backend.shared.utils import count_tokens @@ -1236,6 +1394,14 @@ async def validate_removal( temperature=0.0, # Deterministic removal validation max_tokens=rag_config.validator_max_output_tokens ) + except ProviderContextLengthError as api_error: + raise ContextAllocationError.from_provider_error( + api_error, + "Validator context overflow or provider context mismatch during cleanup removal validation: " + "the provider rejected the mandatory validation prompt as too large.", + context_window=context_allocator.validator_context_window, + output_reserve=rag_config.validator_max_output_tokens, + ) from api_error except Exception as api_error: logger.error(f"CLEANUP DEBUG: API call failed: {api_error}") # Notify task completed even on failure @@ -1275,9 +1441,24 @@ async def validate_removal( logger.warning(f"CLEANUP DEBUG: JSON PARSE FAILED: {e}") logger.warning(f"CLEANUP DEBUG: Raw output that failed to parse:\n{llm_output}") logger.warning(f"Removal validation: JSON parse failed: {e}") - return False - + parsed = await self._retry_validator_json_parse( + prompt, llm_output, task_id + ) + if parsed is None: + return False decision = parsed.get("decision", "reject") + if decision not in {"accept", "reject"}: + logger.warning("Removal validation: invalid decision %r", decision) + return False + await enqueue_optional_update( + parsed, + self.solution_path_manager, + proposer_role=self.role_id, + source_task_id=task_id, + source_phase="cleanup_removal_validation", + source_decision=decision, + ) + reasoning = parsed.get("reasoning", "") logger.info(f"CLEANUP DEBUG: Parsed fields - decision={decision}") @@ -1298,7 +1479,7 @@ async def validate_removal( ) return False - except FreeModelExhaustedError: + except (FreeModelExhaustedError, ContextAllocationError): raise except RetryableProviderError: raise diff --git a/backend/aggregator/core/chroma_cache.py b/backend/aggregator/core/chroma_cache.py new file mode 100644 index 0000000..aaa3684 --- /dev/null +++ b/backend/aggregator/core/chroma_cache.py @@ -0,0 +1,271 @@ +""" +Safe maintenance for ChromaDB's persistent on-disk cache. + +Chroma collections are rebuildable indexes over MOTO's durable text/JSON +sources. Some Chroma versions can leave orphaned UUID segment directories after +collection deletion; this module detects that specific buildup and resets only +the Chroma cache directory. +""" +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime, timezone +import json +import logging +import os +from pathlib import Path +import re +import shutil +import sqlite3 +from typing import Iterable + +logger = logging.getLogger(__name__) + +_UUID_DIR_RE = re.compile( + r"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-" + r"[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$" +) +_MIN_ORPHAN_UUID_DIRS_FOR_RESET = 64 +_MIN_ORPHAN_RATIO_FOR_RESET = 0.75 +_REBUILD_MARKER_SUFFIX = ".rebuild.json" +_QUARANTINE_TOKEN = ".quarantine-" + + +@dataclass(frozen=True) +class ChromaCacheMaintenanceResult: + """Outcome of a Chroma cache maintenance pass.""" + + checked: bool + reset_performed: bool + reason: str = "" + uuid_dir_count: int = 0 + unreferenced_uuid_dir_count: int = 0 + + +def _safe_resolve_chroma_cache_dir(chroma_dir: str | Path, data_dir: str | Path) -> Path: + """Resolve and validate the managed Chroma cache directory.""" + data_root = Path(data_dir).resolve() + cache_dir = Path(chroma_dir).resolve() + + data_root_real = os.path.realpath(os.path.normpath(str(data_root))) + cache_dir_real = os.path.realpath(os.path.normpath(str(cache_dir))) + data_prefix = data_root_real if data_root_real.endswith(os.sep) else data_root_real + os.sep + + if cache_dir_real == data_root_real or not cache_dir_real.startswith(data_prefix): + raise ValueError("Chroma cache directory must be a child of the configured data root") + + if cache_dir.parent == cache_dir: + raise ValueError("Refusing to manage filesystem root as Chroma cache directory") + + return Path(cache_dir_real) + + +def _uuid_directories(cache_dir: Path) -> set[str]: + if not cache_dir.exists(): + return set() + return {child.name for child in cache_dir.iterdir() if child.is_dir() and _UUID_DIR_RE.match(child.name)} + + +def _referenced_chroma_ids(sqlite_path: Path) -> set[str]: + """Read live Chroma collection/segment IDs from SQLite metadata.""" + referenced: set[str] = set() + con = sqlite3.connect(f"file:{sqlite_path}?mode=ro", uri=True) + try: + cur = con.cursor() + for table in ("collections", "segments"): + try: + rows: Iterable[tuple] = cur.execute(f"select * from {table}").fetchall() + except sqlite3.Error: + continue + for row in rows: + for value in row: + if isinstance(value, str) and _UUID_DIR_RE.match(value): + referenced.add(value) + finally: + con.close() + return referenced + + +def _rebuild_marker(cache_dir: Path) -> Path: + return cache_dir.parent / f"{cache_dir.name}{_REBUILD_MARKER_SUFFIX}" + + +def _validated_quarantine_path(cache_dir: Path, candidate: Path) -> Path: + resolved = candidate.resolve() + if resolved.parent != cache_dir.parent or not resolved.name.startswith( + f"{cache_dir.name}{_QUARANTINE_TOKEN}" + ): + raise ValueError("Invalid Chroma quarantine path") + return resolved + + +def recover_interrupted_chroma_rebuild( + chroma_dir: str | Path, + data_dir: str | Path, +) -> None: + """Recover a prior atomic cache quarantine without touching durable sources.""" + cache_dir = _safe_resolve_chroma_cache_dir(chroma_dir, data_dir) + marker = _rebuild_marker(cache_dir) + if not marker.exists(): + return + try: + payload = json.loads(marker.read_text(encoding="utf-8")) + quarantine = _validated_quarantine_path(cache_dir, Path(payload["quarantine"])) + except Exception: + logger.exception("Invalid Chroma rebuild marker at %s; leaving it for operator review", marker) + return + + # A fresh cache directory means the atomic switch completed. The old cache + # is derived data and can be removed. If no fresh directory exists, restore + # the quarantined cache so startup never silently loses the last usable index. + try: + if cache_dir.exists(): + if quarantine.exists(): + shutil.rmtree(quarantine) + elif quarantine.exists(): + quarantine.replace(cache_dir) + marker.unlink(missing_ok=True) + except OSError: + logger.warning("Chroma rebuild recovery deferred because cache files are locked", exc_info=True) + + +def quarantine_chroma_cache( + chroma_dir: str | Path, + data_dir: str | Path, +) -> tuple[Path, Path | None]: + """Atomically move the inactive cache aside and create a rebuild marker.""" + cache_dir = _safe_resolve_chroma_cache_dir(chroma_dir, data_dir) + cache_dir.parent.mkdir(parents=True, exist_ok=True) + marker = _rebuild_marker(cache_dir) + quarantine: Path | None = None + if cache_dir.exists(): + stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S%fZ") + quarantine = _validated_quarantine_path( + cache_dir, + cache_dir.parent / f"{cache_dir.name}{_QUARANTINE_TOKEN}{stamp}", + ) + marker.write_text( + json.dumps({"cache": str(cache_dir), "quarantine": str(quarantine)}), + encoding="utf-8", + ) + cache_dir.replace(quarantine) + cache_dir.mkdir(parents=True, exist_ok=True) + return cache_dir, quarantine + + +def complete_chroma_cache_rebuild( + chroma_dir: str | Path, + data_dir: str | Path, + quarantine: Path | None, +) -> None: + """Commit a cache rebuild and remove the inactive quarantine best-effort.""" + cache_dir = _safe_resolve_chroma_cache_dir(chroma_dir, data_dir) + if quarantine is not None: + quarantine = _validated_quarantine_path(cache_dir, quarantine) + try: + if quarantine.exists(): + shutil.rmtree(quarantine) + except OSError: + logger.warning("Could not remove old Chroma quarantine %s", quarantine, exc_info=True) + return + _rebuild_marker(cache_dir).unlink(missing_ok=True) + + +def abort_chroma_cache_rebuild( + chroma_dir: str | Path, + data_dir: str | Path, + quarantine: Path | None, +) -> None: + """Restore the prior inactive cache when fresh native initialization fails.""" + cache_dir = _safe_resolve_chroma_cache_dir(chroma_dir, data_dir) + if quarantine is not None: + quarantine = _validated_quarantine_path(cache_dir, quarantine) + if cache_dir.exists(): + shutil.rmtree(cache_dir) + if quarantine.exists(): + quarantine.replace(cache_dir) + _rebuild_marker(cache_dir).unlink(missing_ok=True) + + +def maintain_chroma_cache_directory( + chroma_dir: str | Path, + data_dir: str | Path, + *, + orphan_threshold: int = _MIN_ORPHAN_UUID_DIRS_FOR_RESET, + orphan_ratio_threshold: float = _MIN_ORPHAN_RATIO_FOR_RESET, +) -> ChromaCacheMaintenanceResult: + """ + Reset Chroma's cache directory only when a clear orphan buildup is detected. + + Durable user/session files live outside this directory. If metadata cannot be + read, this function logs and skips cleanup rather than guessing. + """ + cache_dir = _safe_resolve_chroma_cache_dir(chroma_dir, data_dir) + recover_interrupted_chroma_rebuild(cache_dir, data_dir) + if not cache_dir.exists(): + return ChromaCacheMaintenanceResult(checked=True, reset_performed=False, reason="missing") + + uuid_dirs = _uuid_directories(cache_dir) + if len(uuid_dirs) < orphan_threshold: + return ChromaCacheMaintenanceResult( + checked=True, + reset_performed=False, + reason="below_threshold", + uuid_dir_count=len(uuid_dirs), + ) + + sqlite_path = cache_dir / "chroma.sqlite3" + if not sqlite_path.exists(): + logger.warning( + "Chroma cache has %d UUID directories but no SQLite metadata; skipping automatic reset.", + len(uuid_dirs), + ) + return ChromaCacheMaintenanceResult( + checked=True, + reset_performed=False, + reason="missing_sqlite_metadata", + uuid_dir_count=len(uuid_dirs), + ) + + try: + referenced_ids = _referenced_chroma_ids(sqlite_path) + except sqlite3.Error as exc: + logger.warning("Could not inspect Chroma SQLite metadata; skipping cache reset: %s", exc) + return ChromaCacheMaintenanceResult( + checked=True, + reset_performed=False, + reason="metadata_unreadable", + uuid_dir_count=len(uuid_dirs), + ) + + unreferenced_dirs = uuid_dirs - referenced_ids + orphan_ratio = len(unreferenced_dirs) / max(len(uuid_dirs), 1) + should_reset = ( + len(unreferenced_dirs) >= orphan_threshold + and orphan_ratio >= orphan_ratio_threshold + ) + if not should_reset: + return ChromaCacheMaintenanceResult( + checked=True, + reset_performed=False, + reason="referenced_or_below_ratio", + uuid_dir_count=len(uuid_dirs), + unreferenced_uuid_dir_count=len(unreferenced_dirs), + ) + + logger.warning( + "Resetting Chroma cache at %s after detecting %d unreferenced UUID directories " + "out of %d total UUID directories.", + cache_dir, + len(unreferenced_dirs), + len(uuid_dirs), + ) + _, quarantine = quarantine_chroma_cache(cache_dir, data_dir) + complete_chroma_cache_rebuild(cache_dir, data_dir, quarantine) + return ChromaCacheMaintenanceResult( + checked=True, + reset_performed=True, + reason="orphan_uuid_directories", + uuid_dir_count=len(uuid_dirs), + unreferenced_uuid_dir_count=len(unreferenced_dirs), + ) diff --git a/backend/aggregator/core/context_allocator.py b/backend/aggregator/core/context_allocator.py index fbcc8eb..c727766 100644 --- a/backend/aggregator/core/context_allocator.py +++ b/backend/aggregator/core/context_allocator.py @@ -8,6 +8,7 @@ from backend.shared.config import rag_config from backend.shared.log_redaction import redact_log_text +from backend.shared.provider_errors import ProviderContextLengthError, ProviderRouteIdentity from backend.shared.utils import count_tokens from backend.aggregator.core.rag_manager import rag_manager @@ -25,12 +26,38 @@ def __init__( available_tokens: int | None = None, context_window: int | None = None, output_reserve: int | None = None, + route: ProviderRouteIdentity | None = None, + cause: BaseException | None = None, ) -> None: super().__init__(message) self.required_tokens = required_tokens self.available_tokens = available_tokens self.context_window = context_window self.output_reserve = output_reserve + self.route = route + self.cause = cause + + @classmethod + def from_provider_error( + cls, + error: ProviderContextLengthError, + message: str, + *, + required_tokens: int | None = None, + available_tokens: int | None = None, + context_window: int | None = None, + output_reserve: int | None = None, + ) -> "ContextAllocationError": + """Translate provider overflow while preserving its effective route.""" + return cls( + message, + required_tokens=required_tokens, + available_tokens=available_tokens, + context_window=context_window, + output_reserve=output_reserve, + route=error.route, + cause=error, + ) class ContextAllocator: diff --git a/backend/aggregator/core/coordinator.py b/backend/aggregator/core/coordinator.py index a9c278c..1743697 100644 --- a/backend/aggregator/core/coordinator.py +++ b/backend/aggregator/core/coordinator.py @@ -22,6 +22,7 @@ CONTEXT_OVERFLOW_RESOLUTION, CONTEXT_OVERFLOW_STOP_MESSAGE, CONTEXT_OVERFLOW_STOP_REASON, + context_overflow_model_payload, ) from backend.aggregator.agents.submitter import SubmitterAgent from backend.aggregator.agents.validator import ValidatorAgent @@ -89,9 +90,6 @@ def __init__(self): self.validator: Optional[ValidatorAgent] = None self.is_running = False - # Stats file path - self.stats_file_path = Path(system_config.data_dir) / "aggregator_stats.json" - # Stats self.total_submissions = 0 self.total_acceptances = 0 @@ -109,6 +107,7 @@ def __init__(self): self._validator_task: Optional[asyncio.Task] = None self._main_task: Optional[asyncio.Task] = None # For single-model mode self._rechunk_task: Optional[asyncio.Task] = None + self._rechunk_follow_up_requested = False self._rechunk_callback_set = False # Chunk size cycling for incremental re-chunking @@ -141,12 +140,27 @@ def __init__(self): self.persist_event_log = True self.fatal_error_message: Optional[str] = None self.fatal_error_type: Optional[str] = None + self.fatal_error_payload: Optional[Dict[str, Any]] = None # Optional source-level hard cap used by autonomous brainstorm mode. self.max_total_acceptances: Optional[int] = None self.acceptance_count_offset: int = 0 self.acceptance_cap_callback: Optional[Callable[[int], Any]] = None self._acceptance_cap_reached = False + self.top_level_terminal_callback: Optional[Callable[[], Any]] = None + + def _notify_top_level_terminal(self) -> None: + """Notify a route-owned top-level lifecycle without affecting child coordinators.""" + if self.top_level_terminal_callback: + try: + self.top_level_terminal_callback() + except Exception: + logger.exception("Aggregator top-level terminal callback failed") + + @property + def stats_file_path(self) -> Path: + """Resolve manual Aggregator stats against the active runtime root.""" + return Path(system_config.data_dir) / "aggregator_stats.json" async def _load_stats(self) -> None: """Load persisted stats from file.""" @@ -244,6 +258,8 @@ async def initialize( trusted_context_texts: Optional[Dict[str, str]] = None, proof_database_store: Optional[Any] = None, local_rejection_log_dir: Optional[str] = None, + solution_path_manager: Optional[Any] = None, + solution_path_acceptance_count_owner: bool = True, local_rejection_log_template: Optional[str] = None, reset_local_rejection_logs_on_start: bool = False, assistant_workflow_mode_override: Optional[str] = None, @@ -277,6 +293,10 @@ async def initialize( local_rejection_log_dir: Optional directory for submitter rejection logs. Internal child aggregators use this to avoid sharing the manual Aggregator rejection files. + solution_path_acceptance_count_owner: Whether this coordinator may + advance the shared path activation count. Autonomous child + aggregators receive the manager for context/proposals while + the autonomous parent remains the sole count owner. local_rejection_log_template: Optional filename template containing ``{submitter_id}``. reset_local_rejection_logs_on_start: Clear the scoped local @@ -291,6 +311,8 @@ async def initialize( self.acceptance_count_offset = max(0, acceptance_count_offset) self.acceptance_cap_callback = acceptance_cap_callback self._acceptance_cap_reached = False + self.solution_path_manager = solution_path_manager + self.solution_path_acceptance_count_owner = solution_path_acceptance_count_owner self.persist_event_log = not skip_stats_load # Validate submitter count @@ -397,7 +419,7 @@ async def initialize( # CRITICAL: Clear RAG for manual mode to prevent cross-contamination # from autonomous brainstorm content that may have been loaded in a prior session logger.info("Clearing RAG for fresh Part 1 aggregator session...") - await asyncio.to_thread(rag_manager.clear_all_documents) + await rag_manager.clear_all_documents_async() logger.info("RAG cleared successfully for Part 1 aggregator") await self._rebuild_shared_training_rag_after_cleanup() logger.info("Persisted Part 1 accepted submissions re-indexed after RAG clear") @@ -471,6 +493,7 @@ async def initialize( local_rejection_log_template=local_rejection_log_template, reset_local_rejection_log_on_initialize=reset_local_rejection_logs_on_start, assistant_workflow_mode_override=assistant_workflow_mode_override, + solution_path_manager=solution_path_manager, ) await submitter.initialize() # Set callback to add submissions to queue @@ -508,6 +531,7 @@ async def initialize( user_files_content=user_files_content, websocket_broadcaster=self.websocket_broadcaster, proof_database_store=proof_database_store, + solution_path_manager=solution_path_manager, ) await self.validator.initialize() # Set task tracking callback for workflow panel integration @@ -725,34 +749,48 @@ async def start(self) -> None: self.fatal_error_type = None logger.info("Starting coordinator...") - # Reset free model manager state for fresh start - free_model_manager.reset() - - # Refresh workflow predictions at start - await self.refresh_workflow_predictions() - - if self.single_model_mode: - # Single-model mode: Round-based sequential workflow - logger.info("Starting single-model workflow (sequential submitters + validator)") - self._main_task = asyncio.create_task(self._single_model_workflow()) - else: - # Multi-model mode: Parallel submitters + independent validator - logger.info("Starting multi-model workflow (parallel submitters)") + try: + free_model_manager.reset() + await self.refresh_workflow_predictions() + + if self.single_model_mode: + logger.info("Starting single-model workflow (sequential submitters + validator)") + self._main_task = asyncio.create_task(self._single_model_workflow()) + else: + logger.info("Starting multi-model workflow (parallel submitters)") + for submitter in self.submitters: + await submitter.start() + self._validator_task = asyncio.create_task(self._validator_loop()) + + await self._broadcast("system_started", {"message": "Aggregator system started"}) + await self._add_persisted_event( + "system_started", + "Aggregator system started", + {"single_model_mode": self.single_model_mode, "submitter_count": len(self.submitters)}, + ) + logger.info("Coordinator started successfully") + except BaseException: + self.is_running = False for submitter in self.submitters: - await submitter.start() - self._validator_task = asyncio.create_task(self._validator_loop()) - - await self._broadcast("system_started", {"message": "Aggregator system started"}) - await self._add_persisted_event( - "system_started", - "Aggregator system started", - {"single_model_mode": self.single_model_mode, "submitter_count": len(self.submitters)}, - ) - logger.info("Coordinator started successfully") + try: + await submitter.stop() + except Exception: + logger.exception("Failed to roll back submitter startup") + for task in (self._main_task, self._validator_task): + if task and not task.done(): + await _cancel_and_drain_task(task) + self._main_task = None + self._validator_task = None + raise async def stop(self) -> None: """Stop the aggregator system.""" - if not self.is_running: + tasks_alive = any( + task is not None and not task.done() + for task in (self._main_task, self._validator_task, self._rechunk_task) + ) + if not self.is_running and not tasks_alive: + self._notify_top_level_terminal() return self.is_running = False @@ -762,6 +800,7 @@ async def stop(self) -> None: # Single-model mode: Cancel main task if self._main_task: await _cancel_and_drain_task(self._main_task) + self._main_task = None else: # Multi-model mode: Stop submitters and validator task for submitter in self.submitters: @@ -769,11 +808,13 @@ async def stop(self) -> None: if self._validator_task: await _cancel_and_drain_task(self._validator_task) + self._validator_task = None # Cancel re-chunking task if running if self._rechunk_task and not self._rechunk_task.done(): logger.info("Cancelling background re-chunking task...") await _cancel_and_drain_task(self._rechunk_task) + self._rechunk_task = None # The queue manager is process-global. Clear it on stop so submissions # from a stopped mini-aggregator cannot be validated under a later phase. @@ -786,6 +827,7 @@ async def stop(self) -> None: {"total_acceptances": self.total_acceptances, "total_rejections": self.total_rejections}, ) logger.info("Coordinator stopped") + self._notify_top_level_terminal() async def add_submission_to_queue(self, submission: Submission) -> None: """Add a submission to the queue (called by submitters).""" @@ -1011,6 +1053,14 @@ async def _handle_acceptance(self, submission: Submission, result: ValidationRes self.total_acceptances += 1 total_acceptances_with_offset = self.acceptance_count_offset + self.total_acceptances + if ( + self.solution_path_manager is not None + and self.solution_path_acceptance_count_owner + ): + from backend.shared.solution_path.integration import note_acceptances + await note_acceptances( + self.solution_path_manager, total_acceptances_with_offset + ) # Add to shared training await shared_training_memory.add_accepted_submission(submission.content) @@ -1090,6 +1140,7 @@ async def _handle_acceptance_cap_reached(self, total_acceptances: int) -> None: self._acceptance_cap_reached = True self.is_running = False + self._notify_top_level_terminal() logger.info( "Acceptance cap reached at %s total acceptances; stopping aggregator at source", @@ -1270,6 +1321,7 @@ async def _handle_context_overflow(self, error: ContextAllocationError, *, role_ self.fatal_error_type = "context_overflow" self.fatal_error_message = str(error) self.is_running = False + self._notify_top_level_terminal() logger.error("Fatal context overflow in %s: %s", role_id, error) @@ -1290,8 +1342,16 @@ async def _handle_context_overflow(self, error: ContextAllocationError, *, role_ await queue_manager.clear() + config_role_id = role_id + if role_id == "aggregator_single_model": + config_role_id = "aggregator_validator" payload = { + "workflow_mode": "aggregator" if self.persist_event_log else "autonomous", "role_id": role_id, + **context_overflow_model_payload( + api_client_manager.get_role_config(config_role_id), + route=getattr(error, "route", None), + ), "reason": CONTEXT_OVERFLOW_STOP_REASON, "message": CONTEXT_OVERFLOW_STOP_MESSAGE, "error_detail": str(error), @@ -1301,6 +1361,7 @@ async def _handle_context_overflow(self, error: ContextAllocationError, *, role_ "output_reserve": getattr(error, "output_reserve", None), "resolution": CONTEXT_OVERFLOW_RESOLUTION, } + self.fatal_error_payload = dict(payload) await self._broadcast("context_overflow_error", payload) await self._add_persisted_event("context_overflow_error", payload["message"], payload) @@ -1467,12 +1528,10 @@ async def _perform_cleanup_review(self) -> None: async def _on_training_update(self) -> None: """Callback when shared training is updated - trigger NON-BLOCKING re-chunking.""" - # Cancel previous re-chunking task if still running - # CRITICAL: Don't await the cancellation - that would block the validator loop! if self._rechunk_task and not self._rechunk_task.done(): - logger.warning("Previous re-chunking still in progress, cancelling it...") - self._rechunk_task.cancel() - # Task will catch CancelledError and clean up in background + self._rechunk_follow_up_requested = True + logger.info("Coalesced training update behind the active RAG commit") + return # Launch re-chunking in background task self._rechunk_task = asyncio.create_task(self._rechunk_training_data()) @@ -1481,24 +1540,22 @@ async def _on_training_update(self) -> None: async def _rechunk_training_data(self) -> None: """Background task for incremental re-chunking training data with global lock.""" try: - # ACQUIRE GLOBAL RAG LOCK - await rag_operation_lock.acquire("Aggregator immediate re-chunk") - - logger.info("Background incremental re-chunking started...") + async with rag_operation_lock.operation("Aggregator immediate re-chunk"): + logger.info("Background incremental re-chunking started...") # Get only new submissions since last RAG - new_submissions = await shared_training_memory.get_new_submissions_since_last_rag() + new_submissions = await shared_training_memory.get_new_submissions_since_last_rag() - if not new_submissions: - logger.info("Incremental re-chunking: No new submissions to process") - await self._broadcast("rechunk_complete", { - "chunk_size": None, - "new_submissions": 0, - "mode": "incremental" - }) - return + if not new_submissions: + logger.info("Incremental re-chunking: No new submissions to process") + await self._broadcast("rechunk_complete", { + "chunk_size": None, + "new_submissions": 0, + "mode": "incremental" + }) + return - logger.info(f"Incremental re-chunking: Processing {len(new_submissions)} new submissions") + logger.info(f"Incremental re-chunking: Processing {len(new_submissions)} new submissions") # DESIGN NOTE: Incremental re-chunking intentionally accumulates chunks over time. # Each batch is added with source name "rag_shared_training_update_{chunk_size}". @@ -1509,38 +1566,38 @@ async def _rechunk_training_data(self) -> None: # - Simpler than periodic full re-chunk with no risk of data loss during cleanup # Determine chunk size using coordinator-level cycling - chunk_size = rag_config.submitter_chunk_intervals[self.current_rechunk_index] - self.current_rechunk_index = (self.current_rechunk_index + 1) % len(rag_config.submitter_chunk_intervals) - logger.info(f"Incremental re-chunking: Using chunk_size={chunk_size}") + chunk_size = rag_config.submitter_chunk_intervals[self.current_rechunk_index] + self.current_rechunk_index = (self.current_rechunk_index + 1) % len(rag_config.submitter_chunk_intervals) + logger.info(f"Incremental re-chunking: Using chunk_size={chunk_size}") # Add each new submission as text chunks # Combine all new submissions into single text to add - combined_new_content = "\n\n".join([ - f"{'=' * 80}\nSUBMISSION #{sub.get('number') or idx+self.last_ragged_submission_count} | Accepted: {sub.get('timestamp', 'Unknown')}\n{'=' * 80}\n\n{sub['content']}\n" - for idx, sub in enumerate(new_submissions) - ]) + combined_new_content = "\n\n".join([ + f"{'=' * 80}\nSUBMISSION #{sub.get('number') or idx+self.last_ragged_submission_count} | Accepted: {sub.get('timestamp', 'Unknown')}\n{'=' * 80}\n\n{sub['content']}\n" + for idx, sub in enumerate(new_submissions) + ]) # Add the combined new submissions with current chunk size - await rag_manager.add_text( - combined_new_content, - f"rag_shared_training_update_{chunk_size}", # Unique name per chunk size - chunk_sizes=[chunk_size], - is_permanent=False - ) + await rag_manager.add_text( + combined_new_content, + f"rag_shared_training_update_{chunk_size}", + chunk_sizes=[chunk_size], + is_permanent=False + ) # Mark submissions as RAG'd - current_count = await shared_training_memory.get_insights_count() - await shared_training_memory.mark_submissions_ragged(current_count) + current_count = await shared_training_memory.get_insights_count() + await shared_training_memory.mark_submissions_ragged(current_count) - logger.info(f"Incremental re-chunking COMPLETE - {len(new_submissions)} submissions added, chunk_size={chunk_size}") + logger.info(f"Incremental re-chunking COMPLETE - {len(new_submissions)} submissions added, chunk_size={chunk_size}") # Broadcast success - await self._broadcast("rechunk_complete", { - "chunk_size": chunk_size, - "new_submissions": len(new_submissions), - "total_submissions": current_count, - "mode": "incremental" - }) + await self._broadcast("rechunk_complete", { + "chunk_size": chunk_size, + "new_submissions": len(new_submissions), + "total_submissions": current_count, + "mode": "incremental" + }) except asyncio.CancelledError: logger.info("Incremental re-chunking cancelled (newer update triggered)") @@ -1552,8 +1609,9 @@ async def _rechunk_training_data(self) -> None: "message": "Incremental re-chunking failed but system continues" }) finally: - # ALWAYS RELEASE LOCK - rag_operation_lock.release() + if self._rechunk_follow_up_requested and self.is_running: + self._rechunk_follow_up_requested = False + self._rechunk_task = asyncio.create_task(self._rechunk_training_data()) async def _rebuild_shared_training_rag_after_cleanup(self) -> None: """Full RAG rebuild of shared-training content after a cleanup removal. @@ -1566,8 +1624,7 @@ async def _rebuild_shared_training_rag_after_cleanup(self) -> None: current_path = Path(shared_training_memory.file_path) current_count = await shared_training_memory.get_insights_count() - await rag_operation_lock.acquire("Aggregator cleanup full re-rag") - try: + async with rag_operation_lock.operation("Aggregator cleanup full re-rag"): # Collect every source name that could contain shared-training chunks candidate_sources = [current_path.name, current_path.with_suffix(".tmp").name] for size in rag_config.submitter_chunk_intervals: @@ -1586,11 +1643,6 @@ async def _rebuild_shared_training_rag_after_cleanup(self) -> None: await shared_training_memory.mark_submissions_ragged(current_count) logger.info(f"Cleanup full re-RAG complete: {current_count} live submissions re-indexed") - except Exception as e: - logger.error(f"Cleanup full re-RAG failed: {e}", exc_info=True) - raise - finally: - rag_operation_lock.release() async def get_status(self) -> SystemStatus: """Get current system status.""" @@ -1667,7 +1719,7 @@ async def clear_all_submissions(self) -> None: # Clear RAG database try: - await asyncio.to_thread(rag_manager.clear_all_documents) + await rag_manager.clear_all_documents_async() logger.info("Cleared RAG database") except Exception as e: logger.error(f"Failed to clear RAG database: {e}") diff --git a/backend/aggregator/core/rag_manager.py b/backend/aggregator/core/rag_manager.py index 81dc012..58dc136 100644 --- a/backend/aggregator/core/rag_manager.py +++ b/backend/aggregator/core/rag_manager.py @@ -12,6 +12,8 @@ import logging import hashlib import time +import gc +import platform from pathlib import Path from backend.shared.config import rag_config, system_config @@ -21,8 +23,15 @@ from backend.shared.utils import count_tokens from backend.shared.log_redaction import redact_log_text from backend.aggregator.ingestion.pipeline import ingestion_pipeline +from backend.aggregator.core.chroma_cache import ( + abort_chroma_cache_rebuild, + complete_chroma_cache_rebuild, + maintain_chroma_cache_directory, + quarantine_chroma_cache, +) logger = logging.getLogger(__name__) +CHROMA_DELETE_BATCH_SIZE = 1000 class RAGManager: @@ -31,20 +40,15 @@ class RAGManager: """ def __init__(self): - # ChromaDB client - self.chroma_client = chromadb.PersistentClient( - path=system_config.chroma_db_dir, - settings=Settings(anonymized_telemetry=False) - ) - - # Collections for different chunk sizes + # Open Chroma lazily so imports are filesystem side-effect free and the + # runtime root can be bound before first use. + self.chroma_client = None self.collections = {} - for size in rag_config.submitter_chunk_intervals: - collection_name = f"chunks_{size}" - self.collections[size] = self.chroma_client.get_or_create_collection( - name=collection_name, - metadata={"chunk_size": size} - ) + self._root_identity = None + self._prepared_root_identities = set() + # Rust collection/client wrappers can retain Windows directory handles + # until their finalizers run even after the shared system is stopped. + gc.collect() # In-memory chunk storage for BM25 self.chunks_by_size: Dict[int, List[DocumentChunk]] = { @@ -65,6 +69,223 @@ def __init__(self): self.document_count = 0 self.permanent_documents = set() # User files never evicted self.document_access_order: OrderedDict = OrderedDict() # LRU tracking: source_name -> last_access_time + + @property + def is_initialized(self) -> bool: + return self.chroma_client is not None + + def _prepare_process_generation_cache_locked(self) -> Path | None: + """Replace prior-process Windows Chroma state before Rust can open it.""" + identity = system_config.runtime_root_identity() + if identity in self._prepared_root_identities: + return None + if platform.system() != "Windows" or system_config.generic_mode: + self._prepared_root_identities.add(identity) + return None + + cache_dir = Path(system_config.chroma_db_dir) + if not cache_dir.exists(): + cache_dir.mkdir(parents=True, exist_ok=True) + self._prepared_root_identities.add(identity) + return None + + _, quarantine = quarantine_chroma_cache( + system_config.chroma_db_dir, + system_config.data_dir, + ) + if quarantine is not None: + logger.warning( + "Replaced prior-process Windows Chroma cache before native initialization; " + "RAG sources will be rebuilt lazily from durable files." + ) + return quarantine + + def _ensure_initialized_locked(self) -> None: + """Worker-only open; callers must hold the async lifecycle boundary.""" + identity = system_config.runtime_root_identity() + if self.chroma_client is not None and self._root_identity == identity: + return + if self.chroma_client is not None: + self._close_locked() + self._reset_memory_state() + quarantine = self._prepare_process_generation_cache_locked() + if quarantine is None: + self._maintain_persistent_cache() + try: + self.chroma_client = chromadb.PersistentClient( + path=system_config.chroma_db_dir, + settings=Settings(anonymized_telemetry=False), + ) + self.collections = { + size: self.chroma_client.get_or_create_collection( + name=f"chunks_{size}", + metadata={"chunk_size": size}, + ) + for size in rag_config.submitter_chunk_intervals + } + self._root_identity = identity + except BaseException: + self._close_locked() + try: + abort_chroma_cache_rebuild( + system_config.chroma_db_dir, + system_config.data_dir, + quarantine, + ) + finally: + self._prepared_root_identities.discard(identity) + raise + else: + self._prepared_root_identities.add(identity) + if quarantine is not None: + complete_chroma_cache_rebuild( + system_config.chroma_db_dir, + system_config.data_dir, + quarantine, + ) + + async def ensure_initialized(self) -> None: + async with rag_operation_lock.operation("Chroma initialize"): + await self._await_native_worker( + "Chroma initialize", + self._ensure_initialized_locked, + ) + + async def prepare_process_generation_cache(self) -> None: + """Prepare the active cache before any workflow can enter Chroma.""" + async with rag_operation_lock.operation("Chroma process-generation prepare"): + await self._await_native_worker( + "Chroma process-generation prepare", + self._ensure_initialized_locked, + ) + + async def _await_native_worker(self, operation_name: str, func, *args, **kwargs): + """Run one native call and drain it before propagating cancellation.""" + worker = asyncio.ensure_future(asyncio.to_thread(func, *args, **kwargs)) + try: + return await asyncio.shield(worker) + except asyncio.CancelledError: + current = asyncio.current_task() + uncancel = getattr(current, "uncancel", None) + pending_cancellations = 0 + if callable(uncancel): + while current is not None and current.cancelling(): + uncancel() + pending_cancellations += 1 + try: + try: + await asyncio.shield(worker) + except Exception: + logger.debug( + "Chroma worker failed while completing cancelled operation %s", + operation_name, + exc_info=True, + ) + finally: + for _ in range(pending_cancellations): + if current is not None: + current.cancel() + raise + + async def _run_chroma_call( + self, + operation_name: str, + chunk_size: int, + method_name: str, + *args, + **kwargs, + ): + """Run one Chroma call serially and never abandon its native worker. + + Cancelling ``asyncio.to_thread`` only cancels the awaiter; the native + Chroma/Rust call keeps running. Waiting for that worker before releasing + the global lock prevents a subsequent reset/query from entering Chroma + concurrently with the abandoned call. + """ + async with rag_operation_lock.operation(operation_name): + await self._await_native_worker( + f"{operation_name} initialization", + self._ensure_initialized_locked, + ) + collection = self.collections[chunk_size] + return await self._await_native_worker( + operation_name, + getattr(collection, method_name), + *args, + **kwargs, + ) + + def _close_locked(self) -> None: + """Release the current Chroma client without touching durable data.""" + client = self.chroma_client + if client is not None: + close_client = getattr(client, "close", None) + stop = getattr(getattr(client, "_system", None), "stop", None) + if callable(close_client): + try: + close_client() + except Exception as exc: + logger.debug("Chroma client close reported: %s", exc) + elif callable(stop): # Compatibility with older supported Chroma. + try: + stop() + except Exception as exc: + logger.debug("Chroma client stop reported: %s", exc) + clear_system_cache = getattr(client, "clear_system_cache", None) + if callable(clear_system_cache): + try: + clear_system_cache() + except Exception as exc: + logger.debug("Chroma shared-system cache cleanup reported: %s", exc) + self.chroma_client = None + self.collections = {} + self._root_identity = None + + def _reset_memory_state(self) -> None: + self.chunks_by_size = {size: [] for size in rag_config.submitter_chunk_intervals} + self.bm25_index = {size: None for size in rag_config.submitter_chunk_intervals} + self.rewrite_cache.clear() + self.bm25_cache.clear() + self.context_pack_cache.clear() + self.document_count = 0 + self.permanent_documents.clear() + self.document_access_order.clear() + + async def close(self) -> None: + """Drain native work, then close the Chroma client exactly once.""" + async with rag_operation_lock.operation("Chroma close"): + await self._await_native_worker("Chroma close", self._close_locked) + + async def reset(self) -> None: + """Close Chroma and clear all process-local retrieval state.""" + async with rag_operation_lock.operation("Chroma reset"): + await self._await_native_worker("Chroma reset", self._close_locked) + self._reset_memory_state() + + def _maintain_persistent_cache(self) -> None: + """Clean orphaned Chroma cache artifacts before opening the client.""" + try: + result = maintain_chroma_cache_directory( + system_config.chroma_db_dir, + system_config.data_dir, + ) + except Exception as exc: + logger.warning("Chroma cache maintenance skipped: %s", exc) + return + + if result.reset_performed: + logger.warning( + "Chroma cache was reset to remove %d orphaned UUID directories; " + "RAG sources will be re-indexed from durable files as workflows start.", + result.unreferenced_uuid_dir_count, + ) + else: + logger.debug( + "Chroma cache maintenance completed without reset: %s (%d UUID dirs, %d unreferenced).", + result.reason, + result.uuid_dir_count, + result.unreferenced_uuid_dir_count, + ) async def add_document( self, @@ -250,49 +471,44 @@ async def _add_chunks(self, chunks: List[DocumentChunk], chunk_size: int) -> Non return texts = [chunk.text for chunk in chunks] + await self.ensure_initialized() + embeddings = await api_client_manager.get_embeddings(texts) - embeddings = None - lock_acquired = False - if system_config.generic_mode: - embeddings = await api_client_manager.get_embeddings(texts) - await rag_operation_lock.acquire(f"RAGManager add_chunks write (size={chunk_size})") - lock_acquired = True - else: - await rag_operation_lock.acquire(f"RAGManager add_chunks (size={chunk_size})") - lock_acquired = True - try: - if embeddings is None: - embeddings = await api_client_manager.get_embeddings(texts) - - # Update chunks with embeddings and tokens - for chunk, embedding in zip(chunks, embeddings): - chunk.embedding = embedding - chunk.tokens = chunk.text.lower().split() + # Update chunks with embeddings and tokens + for chunk, embedding in zip(chunks, embeddings): + chunk.embedding = embedding + chunk.tokens = chunk.text.lower().split() - # ChromaDB writes stay under the global RAG lock in both modes. + # Commit the native write and its process-local mirror under one owner. + operation_name = f"Chroma upsert (size={chunk_size})" + async with rag_operation_lock.operation(operation_name): + await self._await_native_worker( + f"{operation_name} initialization", + self._ensure_initialized_locked, + ) collection = self.collections[chunk_size] try: - await asyncio.to_thread( - collection.add, + await self._await_native_worker( + operation_name, + collection.upsert, ids=[chunk.chunk_id for chunk in chunks], embeddings=embeddings, documents=texts, - metadatas=[chunk.metadata for chunk in chunks] + metadatas=[chunk.metadata for chunk in chunks], ) - logger.debug(f"Added {len(chunks)} chunks to ChromaDB collection (size={chunk_size})") + logger.debug(f"Upserted {len(chunks)} chunks in ChromaDB collection (size={chunk_size})") except Exception as e: - logger.error(f"CRITICAL: ChromaDB add failed for chunk_size={chunk_size}: {type(e).__name__}: {e}") - logger.error(f"Attempting to add {len(chunks)} chunks with IDs: {[c.chunk_id for c in chunks][:5]}...") + logger.error(f"CRITICAL: ChromaDB upsert failed for chunk_size={chunk_size}: {type(e).__name__}: {e}") + logger.error(f"Attempting to upsert {len(chunks)} chunks with IDs: {[c.chunk_id for c in chunks][:5]}...") raise - # Add to memory - self.chunks_by_size[chunk_size].extend(chunks) - - # Invalidate BM25 index for this size + incoming_ids = {chunk.chunk_id for chunk in chunks} + self.chunks_by_size[chunk_size] = [ + existing + for existing in self.chunks_by_size[chunk_size] + if existing.chunk_id not in incoming_ids + ] + chunks self.bm25_index[chunk_size] = None - finally: - if lock_acquired: - rag_operation_lock.release() async def _rewrite_query(self, query: str) -> List[str]: """Stage A: Expand query into semantic variants.""" @@ -334,6 +550,7 @@ async def _hybrid_recall( include_source_prefixes: Optional[List[str]] = None ) -> List[Tuple[DocumentChunk, float]]: """Stage B: Hybrid BM25 + Vector search.""" + await self.ensure_initialized() # Work from a stable snapshot so threaded scoring does not race with # concurrent RAG add/remove operations mutating the live chunk lists. chunks = list(self._filter_chunks_by_source_scope( @@ -379,7 +596,6 @@ async def _vector_search( candidate_chunks: Optional[List[DocumentChunk]] = None ) -> List[Tuple[DocumentChunk, float]]: """Vector similarity search with retry logic for HNSW index race conditions.""" - collection = self.collections[chunk_size] chunks = candidate_chunks if candidate_chunks is not None else self.chunks_by_size[chunk_size] if not chunks: @@ -403,8 +619,10 @@ async def _vector_search( for attempt in range(max_retries): try: - results = await asyncio.to_thread( - collection.query, + results = await self._run_chroma_call( + f"Chroma query (size={chunk_size})", + chunk_size, + "query", query_embeddings=[query_embedding], n_results=min(rag_config.hybrid_recall_top_k, len(chunks)) ) @@ -717,17 +935,21 @@ async def _enforce_chunk_cap(self) -> None: for chunk in chunks: if removed < overflow and not chunk.is_permanent: evict_ids.append(chunk.chunk_id) - chunk.embedding = None removed += 1 else: keep.append(chunk) if evict_ids: - collection = self.collections[chunk_size] try: - await asyncio.to_thread(collection.delete, ids=evict_ids) + await self._run_chroma_call( + f"Chroma chunk-cap delete (size={chunk_size})", + chunk_size, + "delete", + ids=evict_ids, + ) except Exception as e: logger.error(f"ChromaDB delete during chunk cap enforcement (size={chunk_size}): {e}") + raise self.chunks_by_size[chunk_size] = keep self.bm25_index[chunk_size] = None @@ -772,99 +994,93 @@ async def _evict_lru_document(self) -> None: ) async def remove_document(self, source_name: str) -> None: - """Remove a document from all collections.""" - was_tracked = source_name in self.document_access_order - - for chunk_size in rag_config.submitter_chunk_intervals: - # Remove from memory - self.chunks_by_size[chunk_size] = [ - c for c in self.chunks_by_size[chunk_size] - if c.source_file != source_name - ] - - # Remove from ChromaDB - collection = self.collections[chunk_size] - # Get IDs for this source - results = await asyncio.to_thread(collection.get, where={"source_file": source_name}) - if results['ids']: - await asyncio.to_thread(collection.delete, ids=results['ids']) - - # Invalidate BM25 - self.bm25_index[chunk_size] = None - - if was_tracked: - self.document_count = max(0, self.document_count - 1) - - # Clean up LRU tracking - if source_name in self.document_access_order: - del self.document_access_order[source_name] - if source_name in self.permanent_documents: + """Remove a source from every collection before changing memory.""" + async with rag_operation_lock.operation(f"Chroma source removal: {source_name}"): + await self._await_native_worker( + f"Chroma source removal initialization: {source_name}", + self._ensure_initialized_locked, + ) + was_tracked = source_name in self.document_access_order + failures = [] + for chunk_size in rag_config.submitter_chunk_intervals: + try: + await self._delete_matching_ids( + chunk_size, + f"Chroma document delete (size={chunk_size})", + where={"source_file": source_name}, + ) + except Exception as exc: + failures.append(f"chunks_{chunk_size}: {exc}") + if failures: + raise RuntimeError( + f"Failed to remove RAG source {source_name!r}: {'; '.join(failures)}" + ) + + for chunk_size in rag_config.submitter_chunk_intervals: + self.chunks_by_size[chunk_size] = [ + c for c in self.chunks_by_size[chunk_size] + if c.source_file != source_name + ] + self.bm25_index[chunk_size] = None + + if was_tracked: + self.document_count = max(0, self.document_count - 1) + self.document_access_order.pop(source_name, None) self.permanent_documents.discard(source_name) logger.info("Removed document: %s", redact_log_text(source_name, 120)) - - def clear_all_documents(self) -> None: - """Clear all documents from RAG database (synchronous for cleanup). - - Uses graceful degradation: clears what it can even if some operations fail. - Only raises if critical operations (collection creation) fail. - """ - logger.info("Clearing all documents from RAG database...") - - collection_errors = [] - - try: - # Delete all collections (non-critical if individual deletions fail) - for chunk_size in list(self.collections.keys()): - try: - self.chroma_client.delete_collection(f"chunks_{chunk_size}") - logger.info(f"Deleted collection chunks_{chunk_size}") - except Exception as e: - collection_errors.append(f"chunks_{chunk_size}: {e}") - logger.warning(f"Failed to delete collection chunks_{chunk_size}: {e}") - - # Recreate fresh collections (CRITICAL - must succeed) - self.collections = {} - for size in rag_config.submitter_chunk_intervals: - collection_name = f"chunks_{size}" - try: - self.collections[size] = self.chroma_client.get_or_create_collection( - name=collection_name, - metadata={"chunk_size": size} - ) - logger.info(f"Recreated collection {collection_name}") - except Exception as e: - logger.error(f"CRITICAL: Failed to recreate collection {collection_name}: {e}") - raise # Critical failure - cannot continue without collections - - # Clear in-memory storage (safe operations) - self.chunks_by_size = { - size: [] for size in rag_config.submitter_chunk_intervals - } - - # Clear BM25 indices - self.bm25_index = { - size: None for size in rag_config.submitter_chunk_intervals - } - - # Clear caches - self.rewrite_cache.clear() - self.bm25_cache.clear() - self.context_pack_cache.clear() - - # Reset counters - self.document_count = 0 - self.permanent_documents.clear() - self.document_access_order.clear() - - if collection_errors: - logger.warning(f"RAG cleared with {len(collection_errors)} non-critical warnings: {'; '.join(collection_errors)}") - else: - logger.info("Successfully cleared all RAG documents") - - except Exception as e: - logger.error(f"CRITICAL error clearing RAG database: {e}") - raise + + async def _delete_matching_ids(self, chunk_size: int, operation_name: str, *, where=None) -> int: + async with rag_operation_lock.operation(operation_name): + await self._await_native_worker( + f"{operation_name} initialization", + self._ensure_initialized_locked, + ) + collection = self.collections[chunk_size] + deleted = 0 + previous_batch = None + while True: + results = await self._await_native_worker( + f"{operation_name} lookup", + collection.get, + limit=CHROMA_DELETE_BATCH_SIZE, + include=[], + **({"where": where} if where else {}), + ) + ids = tuple(results.get("ids") or ()) + if not ids: + return deleted + if ids == previous_batch: + raise RuntimeError(f"{operation_name} made no progress") + await self._await_native_worker( + operation_name, + collection.delete, + ids=list(ids), + ) + deleted += len(ids) + previous_batch = ids + + async def clear_all_documents_async(self) -> None: + """Atomically replace the rebuildable Chroma cache with an empty cache.""" + async with rag_operation_lock.operation("Chroma cache rebuild"): + await asyncio.to_thread(self._close_locked) + self._reset_memory_state() + cache_dir, quarantine = await asyncio.to_thread( + quarantine_chroma_cache, + system_config.chroma_db_dir, + system_config.data_dir, + ) + try: + await asyncio.to_thread(self._ensure_initialized_locked) + except BaseException: + # Leave the marker/quarantine for deterministic startup recovery. + raise + await asyncio.to_thread( + complete_chroma_cache_rebuild, + cache_dir, + system_config.data_dir, + quarantine, + ) # Global RAG manager instance diff --git a/backend/aggregator/memory/event_log.py b/backend/aggregator/memory/event_log.py index dce385d..d1737e3 100644 --- a/backend/aggregator/memory/event_log.py +++ b/backend/aggregator/memory/event_log.py @@ -23,12 +23,30 @@ class EventLog: """ def __init__(self): - self.file_path = Path(system_config.data_dir) / "aggregator_event_log.txt" self.events: List[Dict[str, Any]] = [] self._lock = asyncio.Lock() + self._root_generation = system_config.runtime_root_generation + self._file_path_override: Path | None = None + + @property + def file_path(self) -> Path: + return self._file_path_override or Path(system_config.data_dir) / "aggregator_event_log.txt" + + @file_path.setter + def file_path(self, value: str | Path) -> None: + """Allow focused tests/custom stores to override without stale default-root capture.""" + self._file_path_override = Path(value) + self._root_generation = system_config.runtime_root_generation + + def _refresh_root(self) -> None: + if self._root_generation != system_config.runtime_root_generation: + self.events = [] + self._file_path_override = None + self._root_generation = system_config.runtime_root_generation async def initialize(self) -> None: """Initialize event log, loading existing events from file.""" + self._refresh_root() self.file_path.parent.mkdir(parents=True, exist_ok=True) if self.file_path.exists(): @@ -65,6 +83,7 @@ async def add_event(self, event_type: str, message: str, metadata: Dict[str, Any metadata: Optional additional data (submitter_id, submission_number, etc.) """ async with self._lock: + self._refresh_root() event = { 'id': len(self.events) + 1, 'type': event_type, @@ -85,11 +104,13 @@ async def add_event(self, event_type: str, message: str, metadata: Dict[str, Any async def get_all_events(self) -> List[Dict[str, Any]]: """Get all events from the log.""" async with self._lock: + self._refresh_root() return list(self.events) async def clear(self) -> None: """Clear all events from the log.""" async with self._lock: + self._refresh_root() self.events = [] try: async with aiofiles.open(self.file_path, 'w', encoding='utf-8') as f: diff --git a/backend/aggregator/memory/shared_training.py b/backend/aggregator/memory/shared_training.py index f01d0e9..cf07c36 100644 --- a/backend/aggregator/memory/shared_training.py +++ b/backend/aggregator/memory/shared_training.py @@ -6,6 +6,7 @@ from pathlib import Path from typing import List, Callable, Optional, Dict import asyncio +import json import logging import re from datetime import datetime @@ -16,6 +17,7 @@ PROOF_APPENDIX_HEADER = "=== PROOFS GENERATED FROM THIS BRAINSTORM (Lean 4 Verified) ===" MANUAL_AGGREGATOR_PROMPT_FILE = "manual_aggregator_prompt.txt" +MANUAL_MAIN_SUBMITTER_CONFIG_FILE = "manual_main_submitter_config.json" def get_manual_aggregator_prompt_path() -> Path: @@ -61,6 +63,42 @@ async def clear_manual_aggregator_prompt() -> None: logger.debug("Unable to clear manual Aggregator prompt: %s", exc) +def get_manual_main_submitter_config_path() -> Path: + return Path(system_config.data_dir) / MANUAL_MAIN_SUBMITTER_CONFIG_FILE + + +async def save_manual_main_submitter_config(config: Dict[str, object]) -> None: + """Persist non-secret Main Submitter 1 settings for backend-restart handoff.""" + path = get_manual_main_submitter_config_path() + path.parent.mkdir(parents=True, exist_ok=True) + temp_path = path.with_name(f"{path.name}.tmp") + async with aiofiles.open(temp_path, "w", encoding="utf-8") as handle: + await handle.write(json.dumps(config, ensure_ascii=False, indent=2)) + await asyncio.to_thread(temp_path.replace, path) + + +async def load_manual_main_submitter_config() -> Dict[str, object]: + path = get_manual_main_submitter_config_path() + if not path.exists(): + return {} + try: + async with aiofiles.open(path, "r", encoding="utf-8") as handle: + payload = json.loads(await handle.read()) + return payload if isinstance(payload, dict) else {} + except (OSError, ValueError, TypeError) as exc: + logger.debug("Unable to load manual Main Submitter config: %s", exc) + return {} + + +async def clear_manual_main_submitter_config() -> None: + path = get_manual_main_submitter_config_path() + if path.exists(): + try: + await asyncio.to_thread(path.unlink) + except FileNotFoundError: + pass + + class SharedTrainingMemory: """ Validator-distributed training database. @@ -68,7 +106,9 @@ class SharedTrainingMemory: """ def __init__(self): - self.file_path = Path(system_config.shared_training_file) + self._file_path = Path(system_config.shared_training_file) + self._uses_default_path = True + self._root_generation = system_config.runtime_root_generation self.insights: List[Dict[str, str]] = [] # Now stores dicts with metadata self.max_insights = rag_config.max_shared_training_insights self.rechunk_callback: Optional[Callable] = None @@ -76,6 +116,32 @@ def __init__(self): self.submission_count = 0 self.last_ragged_submission_count = 0 # Track which submissions have been RAG'd self.proof_appendix = "" + + @property + def file_path(self) -> Path: + if self._root_generation != system_config.runtime_root_generation: + if self._uses_default_path: + self._file_path = Path(system_config.shared_training_file) + self.insights = [] + self.submission_count = 0 + self.last_ragged_submission_count = 0 + self.proof_appendix = "" + else: + system_config.assert_path_in_data_root( + self._file_path, + "shared-training override", + ) + self._root_generation = system_config.runtime_root_generation + return self._file_path + + @file_path.setter + def file_path(self, value: str | Path) -> None: + path = Path(value) + self._file_path = path + self._uses_default_path = path.resolve(strict=False) == Path( + system_config.shared_training_file + ).resolve(strict=False) + self._root_generation = system_config.runtime_root_generation async def initialize(self) -> None: """Initialize shared training memory, creating file if needed.""" diff --git a/backend/aggregator/prompts/submitter_prompts.py b/backend/aggregator/prompts/submitter_prompts.py index 0af19f6..3b7d833 100644 --- a/backend/aggregator/prompts/submitter_prompts.py +++ b/backend/aggregator/prompts/submitter_prompts.py @@ -3,14 +3,15 @@ """ -EMPIRICAL_PROVENANCE_RULES = """EMPIRICAL PROVENANCE RULES: -- Classify concrete claims as one of: theoretical claim, literature claim, empirical claim, or artifact claim. -- Theoretical claims must be supported by sound reasoning, derivation, proof sketch, or explicit assumptions. -- Literature claims must name the external source in-text; never rely on vague phrases like "studies show" or "prior work proves" without identifying the source. +EMPIRICAL_PROVENANCE_RULES = """CLAIM-TYPE RIGOR AND PROVENANCE RULES: +- Match the verification standard to the claim type and domain. Novelty never overrides correctness, provenance, safety, or honesty. +- Mathematical claims require sound derivation, proof, or explicit assumptions. Strategic or causal claims require valid inference, explicit assumptions, and realistic limitations. +- Literature claims presented as established must identify a source in the response or supplied context; never rely on vague phrases like "studies show" or "prior work proves." If exact attribution is unavailable, state the uncertainty and propose source verification rather than inventing a citation. - Empirical claims include benchmark numbers, latency, throughput, speedup, accuracy, perplexity, hardware performance, ablation outcomes, and measured implementation results. - Artifact claims include statements about code, kernels, logs, experiments, reproductions, or accompanying implementations. - DO NOT present empirical or artifact claims as facts unless they are backed by an explicit external citation or a provided artifact in context. - If such support is absent, rewrite the idea as a hypothesis, design intuition, proposed experiment, expected benefit, or future-work suggestion. +- Engineering and software proposals require a concrete mechanism, relevant constraints, feasibility reasoning, failure modes, and a verification plan. Proposed code, prototypes, or tests must not be described as already implemented. - NEVER invent experiments, benchmark numbers, hardware measurements, datasets, citations, or code artifacts.""" @@ -19,7 +20,7 @@ Only where it is apparent, appearing true, and potentially very helpful, you may use extreme creativity to propose a near-solution or adjacent solution that solves toward the user's prompt and could advance this brainstorm further in future submissions. -Do not force creativity. If the creative route is not apparent or would weaken rigor, submit the strongest normal direct-progress contribution instead.""" +Do not force creativity. Creativity is not permission to fabricate evidence, artifacts, measurements, or certainty. If the creative route is not apparent or would weaken rigor, submit the strongest normal direct-progress contribution instead.""" def get_submitter_system_prompt(lean4_enabled: bool = False) -> str: @@ -51,12 +52,12 @@ def get_submitter_system_prompt(lean4_enabled: bool = False) -> str: if lean4_enabled else "" ) - return ("""You are a mathematical submitter in an AI cluster working to solve complex mathematical problems. Your role is to: + return ("""You are a solution submitter in an AI cluster working to solve the user's exact objective. Your role is to: 1. Analyze the user's prompt and provided context carefully 2. Build upon the shared training database (accepted submissions from other agents) 3. Learn from your rejection history to avoid repeating mistakes -4. Generate novel, valuable mathematical progress that advances the solution +4. Generate the strongest credible and genuinely novel contribution that advances the solution ⚠️ CRITICAL - INTERNAL CONTENT WARNING ⚠️ @@ -71,17 +72,17 @@ def get_submitter_system_prompt(lean4_enabled: bool = False) -> str: """ + EMPIRICAL_PROVENANCE_RULES + """ - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, defensible, and verifiable content under the standards appropriate to its claims. Use internal context as exploration history and your base knowledge for reasoning and verification. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. --- YOUR TASK: -Generate a novel mathematical insight that advances the user's goal. -Generate the strongest rigorous mathematical contribution you can toward the user's goal. Any submission should aggressively address the user's WHOLE question as stated where possible, no partial solutions. +Aggressively pursue the strongest credible and genuinely novel solution to the user's exact objective. +Choose the contribution form and verification standard that fit the problem. Any submission should aggressively address the user's WHOLE question as stated where possible, no partial solutions. -PROGRESSIVE SYSTEM: You will be called MANY times throughout this brainstorming process. Each call should produce ONE deep, well-developed mathematical insight. Do not try to cover everything at once — focus on thoroughly developing a single avenue per submission with full rigor. You will have many more opportunities to explore other avenues in future submissions. +PROGRESSIVE SYSTEM: You will be called MANY times throughout this brainstorming process. Each call should produce ONE deep, well-developed contribution. Do not try to cover everything at once — focus on thoroughly developing a single avenue per submission with full claim-appropriate rigor. You will have many more opportunities to explore other avenues in future submissions. DIRECT-SOLUTION PREFERENCE: - If you can directly answer the user's whole problem, do that FIRST @@ -93,37 +94,37 @@ def get_submitter_system_prompt(lean4_enabled: bool = False) -> str: If the USER PROMPT explicitly says TOPIC EXPLORATION PHASE or PAPER TITLE EXPLORATION PHASE, follow that requested output format exactly: - For TOPIC EXPLORATION PHASE, propose one candidate brainstorm question optimized to directly answer the user's whole prompt if answered, or to answer the next necessary piece when a whole-answer route is not possible in one shot - For PAPER TITLE EXPLORATION PHASE, propose one candidate paper title optimized for communicating the paper's direct answer-bearing content -- In these meta-phases, do NOT solve the mathematical problem or write the paper unless the user prompt explicitly asks for that; the direct-solution preference means the candidate should point toward or communicate direct resolution +- In these meta-phases, do NOT solve the underlying problem or write the paper unless the user prompt explicitly asks for that; the direct-solution preference means the candidate should point toward or communicate direct resolution -Focus on mathematical concepts, theorems, techniques, and proofs that directly answer the mathematical problem in the prompt whenever possible. Use all available resources including web search if available. +Use the solution form that best fits the objective: a complete answer, invention, mechanism, design, algorithm, mathematical result, theorem, proof, formalization, experimental proposal, falsifiable hypothesis, counterexample, impossibility argument, implementation strategy, or risk analysis. Mathematical reasoning and formal proof remain first-class methods whenever relevant. Use available verification resources, including web search when available. WHAT MAKES A VALUABLE SUBMISSION - Consider: - Does it directly answer the user's whole problem, or where that is not realistic in one step, a necessary piece of it? - Does it add genuinely new information or perspectives beyond what is already in the training database? -- Does it connect existing mathematical concepts in novel ways? -- Does it provide concrete methods, theorems, proofs, or mathematical techniques? +- Does it provide a concrete mechanism, method, design, algorithm, theorem, proof, experiment, or other objective-appropriate contribution? - Is it specific and actionable, not vague or generic? -- Does it increase solution availability or narrow the search space? -- Is it based on established mathematical principles and rigorous logic? +- Does it identify relevant assumptions, constraints, feasibility limits, failure modes, and ways to verify or falsify the proposal? +- Does it increase solution availability or narrow the search space while remaining correct and defensible? CRITICAL REQUIREMENTS - CONTENT: -- ALL submissions must be rooted in sound mathematical reasoning - NO unfounded claims or logical fallacies +- ALL submissions must meet domain-appropriate standards of correctness and defensibility - NO unfounded claims, fabricated support, or logical fallacies - Prefer directly resolving the user's whole problem over auxiliary exposition - Piecewise submissions are acceptable only when the piece is a clearly necessary step toward the full answer, not because it is easier or merely adjacent -- Focus on mathematical concepts, theorems, and techniques that are verifiable and established +- Mathematics, theorem discovery, proof, and formalization are explicitly welcome when relevant; mathematical claims require sound derivation, proof, or explicit assumptions +- Engineering and software proposals require mechanisms, constraints, feasibility reasoning, failure modes, and verification plans - Be specific and actionable, not vague or generic - Avoid redundancy with existing accepted submissions - Focus on increasing solution availability or narrowing the search space -- Present rigorous mathematical arguments - Unsupported empirical or artifact claims must be framed as proposals, hypotheses, or future work rather than as completed results Your submission will be validated against these criteria: - Does it provide the strongest direct progress currently justified? - Does it meaningfully advance the solution space? -- Is it based on sound mathematical principles? +- Is it correct and defensible under the appropriate domain and claim-type standard? - Does it avoid contradictions? - Is it non-redundant with existing knowledge? -- Is it mathematically rigorous? +- Are its provenance, uncertainty, constraints, and verification path honest and sufficiently specific? +- If it makes mathematical claims, are they mathematically rigorous? {lean_proof_route} @@ -132,7 +133,7 @@ def get_submitter_system_prompt(lean4_enabled: bool = False) -> str: Normal brainstorm idea: { "submission_type": "idea", - "submission": "Your detailed mathematical submission describing concepts, theorems, proofs, and approaches based on established mathematical principles.", + "submission": "Your detailed contribution in the form best suited to the objective, with concrete mechanisms, reasoning, assumptions, constraints, evidence, or proof as applicable.", "reasoning": "Brief explanation of why this submission is valuable" } @@ -171,7 +172,7 @@ def get_submitter_json_schema(lean4_enabled: bool = False) -> str: Normal brainstorm idea: { "submission_type": "idea", - "submission": "string - your detailed mathematical submission with theorems, proofs, and techniques", + "submission": "string - your detailed, credible contribution using the solution form and verification standard appropriate to the objective", "reasoning": "string - explanation of submission value" } {lean_proof_schema} @@ -194,11 +195,18 @@ def get_submitter_json_schema(lean4_enabled: bool = False) -> str: "reasoning": "This submission provides the rigorous mathematical foundation for why squaring the circle is impossible, connecting transcendental number theory to geometric constructibility." } -GOOD Example (technique application): +GOOD Example (engineering mechanism): { "submission_type": "idea", - "submission": "For problems involving irrational approximations, continued fractions provide optimal rational approximations. The continued fraction expansion of \\\\pi = [3; 7, 15, 1, 292, ...] shows that 22/7 and 355/113 are best rational approximants within their denominator ranges. This technique generalizes: for any irrational \\\\alpha, its convergents p_n/q_n satisfy |\\\\alpha - p_n/q_n| < 1/(q_n * q_{n+1}), providing provably good approximations.", - "reasoning": "Leverages established number theory techniques for understanding irrational approximations relevant to the mathematical problem." + "submission": "Use a segmented battery pack with cell-level voltage and temperature sensing, redundant contactors, and a controller that isolates a segment when rate-of-change thresholds indicate thermal runaway. The design trades added mass and contact resistance for fault containment. Validate it with hardware-in-the-loop fault injection, then instrumented abuse tests; likely failure modes include sensor drift, welded contactors, and propagation faster than isolation.", + "reasoning": "Provides a concrete safety mechanism, constraints, failure modes, and a staged verification plan without claiming that a prototype or test result already exists." +} + +GOOD Example (proposed empirical test): +{ + "submission_type": "idea", + "submission": "Test whether retrieval freshness, rather than model size, causes the observed support failures by holding the generator fixed and randomizing index age across otherwise matched queries. Pre-register freshness buckets, answer-quality metrics, and the null hypothesis; report the result only after collecting data.", + "reasoning": "Offers a falsifiable experiment and clearly treats its outcome as unknown rather than fabricating measurements." } {lean_proof_note} diff --git a/backend/aggregator/prompts/validator_prompts.py b/backend/aggregator/prompts/validator_prompts.py index b6d478d..834abd2 100644 --- a/backend/aggregator/prompts/validator_prompts.py +++ b/backend/aggregator/prompts/validator_prompts.py @@ -3,16 +3,27 @@ """ -EMPIRICAL_PROVENANCE_VALIDATION_RULES = """EMPIRICAL PROVENANCE RULES: -- Classify concrete claims as one of: theoretical claim, literature claim, empirical claim, or artifact claim. -- Theoretical claims must be supported by sound reasoning, derivation, proof sketch, or explicit assumptions. -- Literature claims must identify the external source in-text; vague references like "studies show" are not sufficient. +EMPIRICAL_PROVENANCE_VALIDATION_RULES = """CLAIM-TYPE RIGOR AND PROVENANCE RULES: +- Match the verification standard to the claim type and domain. Novelty never overrides correctness, provenance, safety, or honesty. +- Mathematical claims require sound derivation, proof, or explicit assumptions. Strategic or causal claims require valid inference, explicit assumptions, and realistic limitations. +- Literature claims presented as established must identify a source in the response or supplied context; vague authority phrases are insufficient. If exact attribution is unavailable, require honest uncertainty and proposed source verification rather than an invented citation. - Empirical claims include benchmark numbers, latency, throughput, speedup, accuracy, perplexity, hardware performance, ablations, and measured outcomes. - Artifact claims include statements about code, kernels, experiments, logs, reproductions, or accompanying implementations. - REJECT empirical or artifact claims that are presented as established facts without explicit external citation or a provided artifact in context. - If a submission offers an unsupported benchmark-style idea that is still useful, it must be framed as a proposed experiment, hypothesis, expected benefit, or future-work direction rather than as a completed result. +- Engineering and software proposals must identify a concrete mechanism, relevant constraints, feasibility, failure modes, and a verification plan. Reject claims that proposed code, prototypes, or tests already exist when no artifact is supplied. - NEVER accept invented citations, fabricated experiments, fake benchmark numbers, or nonexistent code artifacts.""" +ALL_PURPOSE_VALIDATION_CRITERIA = """SHARED ALL-PURPOSE EVALUATION CRITERIA: +- Direct impact: Does the contribution directly answer the exact user objective, or target the next necessary piece when a whole answer is not realistic in one submission? +- Genuine novelty: Does it add value beyond accepted memory and obvious or common routes? +- Correctness: Is it defensible under the standards appropriate to its domain and claim type? +- Provenance and uncertainty: Are evidence, sources, artifacts, assumptions, and unknowns represented honestly? +- Specificity and verification: Does it provide an actionable mechanism, method, design, algorithm, theorem, proof, experiment, or other objective-appropriate contribution with a way to verify or falsify it? +- Feasibility: Where relevant, does it address constraints, limitations, risks, and failure modes? +- Non-redundancy: Does it add distinct value rather than restating accepted memory? +- Mathematics remains first-class when relevant. Mathematical claims must be logically sound and supported by derivation, proof, or explicit assumptions; non-mathematical work must not be rejected merely for lacking theorem or proof form.""" + LEAN_VERIFIED_SUBMISSION_RULES = """LEAN 4 VERIFIED SUBMISSION RULES: - A submission containing [LEAN 4 VERIFIED BRAINSTORM PROOF] has already passed Lean 4 and MOTO hard integrity checks before this validator call. - Lean verification only establishes formal validity. It does NOT make a proof useful, novel, or acceptable brainstorm progress. @@ -23,7 +34,7 @@ def get_validator_system_prompt() -> str: """Get system prompt for validator agent.""" - return """You are a validation agent in an AI cluster. Your role is to evaluate mathematical submissions and decide whether they should be added to the shared knowledge base. + return """You are a validation agent in an AI cluster. Your role is to evaluate solution contributions and decide whether they should be added to the shared knowledge base. ⚠️ CRITICAL - INTERNAL CONTENT WARNING ⚠️ @@ -36,9 +47,9 @@ def get_validator_system_prompt() -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content -""" + EMPIRICAL_PROVENANCE_VALIDATION_RULES + "\n\n" + LEAN_VERIFIED_SUBMISSION_RULES + """ +""" + EMPIRICAL_PROVENANCE_VALIDATION_RULES + "\n\n" + ALL_PURPOSE_VALIDATION_CRITERIA + "\n\n" + LEAN_VERIFIED_SUBMISSION_RULES + """ - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to evaluate rigorous, defensible, and verifiable content under the standards appropriate to its claims. Use internal context as exploration history, not as authority. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -47,7 +58,7 @@ def get_validator_system_prompt() -> str: YOUR TASK: Decide whether this submission provides the strongest rigorous progress currently justified toward solving the user's problem, with highest priority given to work that aggressively addresses the user's WHOLE question as stated. -Essentially, you are evaluating whether the knowledge base becomes more useful toward directly answering the user's mathematical prompt with this submission added than it was without it. +Essentially, you are evaluating whether the knowledge base becomes more useful toward directly answering the user's exact objective with this submission added than it was without it. CRITICAL: You are NOT generating solutions yourself. You are judging whether this submission directly answers the whole user question, or where that is not possible in one step, whether it attacks the next best necessary piece better than the current knowledge base does. @@ -61,19 +72,7 @@ def get_validator_system_prompt() -> str: If the USER PROMPT explicitly says TOPIC EXPLORATION PHASE or PAPER TITLE EXPLORATION PHASE, evaluate the submission as the requested candidate artifact, not as a direct solution: - TOPIC EXPLORATION PHASE: accept a candidate brainstorm question if it is specific, distinct, relevant, grounded, and aimed at answering the user's whole prompt if answered, or at the next necessary piece when a whole-answer route is not possible in one shot - PAPER TITLE EXPLORATION PHASE: accept a candidate title if it is accurate, specific, distinct, professional, and foregrounds direct answer-bearing content when justified -- Do NOT reject these meta-phase submissions merely because they are questions or titles rather than mathematical solutions - -EVALUATION CRITERIA - Consider: -- Does the submission directly answer the user's whole problem, or where that is not realistic in one step, a necessary piece of it? -- Does the submission add genuinely new information or perspectives beyond what is already accepted? -- Does the submission connect existing mathematical concepts in novel ways? -- Does the submission provide concrete methods, theorems, proofs, or mathematical techniques? -- Is the submission redundant with current accepted submissions, user provided information, or common mathematical knowledge? -- Is the submission obviously unhelpful or time-wasting content? -- Is the submission grounded in established mathematical principles and rigorous logic? -- Does the submission avoid unfounded claims or logical fallacies? -- Is the submission based on proven mathematical theorems and valid reasoning? -- Are any empirical or artifact claims properly cited or backed by a provided artifact rather than asserted from nowhere? +- Do NOT reject these meta-phase submissions merely because they are questions or titles rather than direct solutions VALIDATION DECISION RULES: A submission should be ACCEPTED if it: @@ -84,17 +83,17 @@ def get_validator_system_prompt() -> str: A submission should be REJECTED if it: 1. Is redundant with the existing accepted submissions -2. Contains trivial or common mathematical knowledge while also having nothing novel to contribute to the knowledge base +2. Contains only trivial or common knowledge and nothing genuinely novel 3. Contains logical contradictions or unsupported claims 4. Is too vague or generic to be actionable 5. Is obviously unhelpful or time-wasting content -6. Contains logical fallacies or mathematically unsound reasoning -7. Presents claims as proven without proper mathematical justification +6. Is incorrect or indefensible under its domain's standards, including mathematically unsound reasoning +7. Presents claims as established without the required evidence, derivation, proof, artifact, or explicit assumptions 8. Presents unsupported empirical, benchmark, hardware, or artifact claims as established fact 9. Is merely tangential or exploratory when a more direct, rigorous contribution was available from the same content 10. Retreats to an easier adjacent/practical/background route while a direct whole-question attack or clearly necessary piecewise attack is available -Ask yourself: "Does adding this submission make us more capable of directly answering the user's mathematical prompt than we were without it, and is this the strongest justified kind of progress?" +Ask yourself: "Does adding this submission make us more capable of directly answering the user's exact objective than we were without it, and is this the strongest justified kind of progress?" REJECTION FEEDBACK FORMAT: If rejecting, your "summary" field must provide CONCRETE, ACTIONABLE guidance using this structure: @@ -208,7 +207,7 @@ def build_validator_prompt( def get_validator_dual_system_prompt() -> str: """Get system prompt for validating TWO submissions simultaneously.""" - return """You are a validation agent in an AI cluster. Your role is to evaluate TWO mathematical submissions simultaneously and decide whether each should be added to the shared knowledge base. + return """You are a validation agent in an AI cluster. Your role is to evaluate TWO solution contributions simultaneously and decide whether each should be added to the shared knowledge base. ⚠️ CRITICAL - INTERNAL CONTENT WARNING ⚠️ @@ -221,9 +220,9 @@ def get_validator_dual_system_prompt() -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content -""" + EMPIRICAL_PROVENANCE_VALIDATION_RULES + "\n\n" + LEAN_VERIFIED_SUBMISSION_RULES + """ +""" + EMPIRICAL_PROVENANCE_VALIDATION_RULES + "\n\n" + ALL_PURPOSE_VALIDATION_CRITERIA + "\n\n" + LEAN_VERIFIED_SUBMISSION_RULES + """ - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to evaluate rigorous, defensible, and verifiable content under the standards appropriate to its claims. Use internal context as exploration history, not as authority. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -235,7 +234,7 @@ def get_validator_dual_system_prompt() -> str: CRITICAL - INDEPENDENT ASSESSMENT: For EACH submission, ask: "Does THIS submission provide the strongest rigorous direct progress currently justified toward the user's whole problem, or the next necessary piece when a whole-answer route is not possible in one shot, considering ONLY the existing database (not the other submission in this batch)?" -Essentially, you are evaluating whether the training database becomes more useful toward directly answering the user's mathematical prompt with each submission added than it was without it. +Essentially, you are evaluating whether the training database becomes more useful toward directly answering the user's exact objective with each submission added than it was without it. DIRECT-SOLUTION PREFERENCE: - Prefer submissions that directly answer the user's whole problem @@ -247,18 +246,7 @@ def get_validator_dual_system_prompt() -> str: If the USER PROMPT explicitly says TOPIC EXPLORATION PHASE or PAPER TITLE EXPLORATION PHASE, evaluate each submission as the requested candidate artifact, not as a direct solution: - TOPIC EXPLORATION PHASE: accept a candidate brainstorm question if it is specific, distinct, relevant, grounded, and aimed at answering the user's whole prompt if answered, or at the next necessary piece when a whole-answer route is not possible in one shot - PAPER TITLE EXPLORATION PHASE: accept a candidate title if it is accurate, specific, distinct, professional, and foregrounds direct answer-bearing content when justified -- Do NOT reject these meta-phase submissions merely because they are questions or titles rather than mathematical solutions - -EVALUATION CRITERIA (Apply to EACH submission independently): -- Does the submission directly answer the user's whole problem, or where that is not realistic in one step, a necessary piece of it? -- Does the submission add genuinely new information or perspectives beyond what is already accepted? -- Does the submission connect existing mathematical concepts in novel ways? -- Does the submission provide concrete methods, theorems, proofs, or mathematical techniques? -- Is the submission redundant with current accepted submissions, user provided information, or common mathematical knowledge? -- Is the submission obviously unhelpful or time-wasting content? -- Is the submission grounded in established mathematical principles and rigorous logic? -- Does the submission avoid unfounded claims or logical fallacies? -- Are any empirical or artifact claims properly cited or backed by a provided artifact rather than asserted from nowhere? +- Do NOT reject these meta-phase submissions merely because they are questions or titles rather than direct solutions VALIDATION DECISION RULES (for each submission): A submission should be ACCEPTED if it: @@ -269,10 +257,10 @@ def get_validator_dual_system_prompt() -> str: A submission should be REJECTED if it: 1. Is redundant with the existing accepted submissions -2. Contains trivial or common mathematical knowledge with nothing novel +2. Contains only trivial or common knowledge with nothing novel 3. Contains logical contradictions or unsupported claims 4. Is too vague or generic to be actionable -5. Contains logical fallacies or mathematically unsound reasoning +5. Is incorrect or indefensible under its domain's standards, including mathematically unsound reasoning 6. Presents unsupported empirical, benchmark, hardware, or artifact claims as established fact 7. Is merely tangential or exploratory when a more direct, rigorous contribution was available from the same content 8. Retreats to an easier adjacent/practical/background route while a direct whole-question attack or clearly necessary piecewise attack is available @@ -365,13 +353,13 @@ def get_validator_dual_json_schema() -> str: { "submission_number": 1, "decision": "accept", - "reasoning": "Submission 1 provides a novel approach to modular arithmetic proofs not in existing database.", + "reasoning": "Submission 1 specifies a segmented battery-isolation mechanism, its sensing thresholds, constraints, and failure modes.", "summary": "" }, { "submission_number": 2, "decision": "accept", - "reasoning": "Submission 2 offers a complementary technique using continued fractions. Not redundant with submission 1.", + "reasoning": "Submission 2 proposes a distinct pre-registered abuse-test protocol that can falsify the isolation design without claiming results already exist.", "summary": "" } ] @@ -383,14 +371,14 @@ def get_validator_dual_json_schema() -> str: { "submission_number": 1, "decision": "accept", - "reasoning": "Submission 1 provides a comprehensive treatment of the Lindemann-Weierstrass theorem with rigorous proofs.", + "reasoning": "Submission 1 provides a complete mathematical argument using the Lindemann-Weierstrass theorem with explicit assumptions and derivation.", "summary": "" }, { "submission_number": 2, "decision": "reject", - "reasoning": "While independently valuable, submission 2 covers the same Lindemann-Weierstrass material as submission 1 but with less rigor. Accepting only submission 1 to prevent redundancy.", - "summary": "Redundant with co-submitted submission 1 which provides more rigorous coverage." + "reasoning": "While independently valuable, submission 2 covers the same Lindemann-Weierstrass argument but omits assumptions already supplied by submission 1. Accepting only submission 1 prevents redundancy.", + "summary": "Redundant with co-submitted submission 1, which provides the more complete mathematical argument." } ] } @@ -446,7 +434,7 @@ def build_validator_dual_prompt( def get_validator_triple_system_prompt() -> str: """Get system prompt for validating THREE submissions simultaneously.""" - return """You are a validation agent in an AI cluster. Your role is to evaluate THREE mathematical submissions simultaneously and decide whether each should be added to the shared knowledge base. + return """You are a validation agent in an AI cluster. Your role is to evaluate THREE solution contributions simultaneously and decide whether each should be added to the shared knowledge base. ⚠️ CRITICAL - INTERNAL CONTENT WARNING ⚠️ @@ -459,9 +447,9 @@ def get_validator_triple_system_prompt() -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content -""" + EMPIRICAL_PROVENANCE_VALIDATION_RULES + "\n\n" + LEAN_VERIFIED_SUBMISSION_RULES + """ +""" + EMPIRICAL_PROVENANCE_VALIDATION_RULES + "\n\n" + ALL_PURPOSE_VALIDATION_CRITERIA + "\n\n" + LEAN_VERIFIED_SUBMISSION_RULES + """ - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to evaluate rigorous, defensible, and verifiable content under the standards appropriate to its claims. Use internal context as exploration history, not as authority. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -473,7 +461,7 @@ def get_validator_triple_system_prompt() -> str: CRITICAL - INDEPENDENT ASSESSMENT: For EACH of the three submissions, ask: "Does THIS submission provide the strongest rigorous direct progress currently justified toward the user's whole problem, or the next necessary piece when a whole-answer route is not possible in one shot, considering ONLY the existing database (not the other submissions in this batch)?" -Essentially, you are evaluating whether the training database becomes more useful toward directly answering the user's mathematical prompt with each submission added than it was without it. +Essentially, you are evaluating whether the training database becomes more useful toward directly answering the user's exact objective with each submission added than it was without it. DIRECT-SOLUTION PREFERENCE: - Prefer submissions that directly answer the user's whole problem @@ -485,18 +473,7 @@ def get_validator_triple_system_prompt() -> str: If the USER PROMPT explicitly says TOPIC EXPLORATION PHASE or PAPER TITLE EXPLORATION PHASE, evaluate each submission as the requested candidate artifact, not as a direct solution: - TOPIC EXPLORATION PHASE: accept a candidate brainstorm question if it is specific, distinct, relevant, grounded, and aimed at answering the user's whole prompt if answered, or at the next necessary piece when a whole-answer route is not possible in one shot - PAPER TITLE EXPLORATION PHASE: accept a candidate title if it is accurate, specific, distinct, professional, and foregrounds direct answer-bearing content when justified -- Do NOT reject these meta-phase submissions merely because they are questions or titles rather than mathematical solutions - -EVALUATION CRITERIA (Apply to EACH submission independently): -- Does the submission directly answer the user's whole problem, or where that is not realistic in one step, a necessary piece of it? -- Does the submission add genuinely new information or perspectives beyond what is already accepted? -- Does the submission connect existing mathematical concepts in novel ways? -- Does the submission provide concrete methods, theorems, proofs, or mathematical techniques? -- Is the submission redundant with current accepted submissions, user provided information, or common mathematical knowledge? -- Is the submission obviously unhelpful or time-wasting content? -- Is the submission grounded in established mathematical principles and rigorous logic? -- Does the submission avoid unfounded claims or logical fallacies? -- Are any empirical or artifact claims properly cited or backed by a provided artifact rather than asserted from nowhere? +- Do NOT reject these meta-phase submissions merely because they are questions or titles rather than direct solutions VALIDATION DECISION RULES (for each submission): A submission should be ACCEPTED if it: @@ -507,10 +484,10 @@ def get_validator_triple_system_prompt() -> str: A submission should be REJECTED if it: 1. Is redundant with the existing accepted submissions -2. Contains trivial or common mathematical knowledge with nothing novel +2. Contains only trivial or common knowledge with nothing novel 3. Contains logical contradictions or unsupported claims 4. Is too vague or generic to be actionable -5. Contains logical fallacies or mathematically unsound reasoning +5. Is incorrect or indefensible under its domain's standards, including mathematically unsound reasoning 6. Presents unsupported empirical, benchmark, hardware, or artifact claims as established fact 7. Is merely tangential or exploratory when a more direct, rigorous contribution was available from the same content 8. Retreats to an easier adjacent/practical/background route while a direct whole-question attack or clearly necessary piecewise attack is available @@ -625,19 +602,19 @@ def get_validator_triple_json_schema() -> str: { "submission_number": 1, "decision": "accept", - "reasoning": "Submission 1 provides a comprehensive proof of the irrationality of sqrt(2) using a novel geometric approach not in existing database.", + "reasoning": "Submission 1 gives a concrete software isolation design with explicit trust boundaries, failure modes, and integration tests not present in the database.", "summary": "" }, { "submission_number": 2, "decision": "reject", - "reasoning": "Submission 2 also addresses sqrt(2) irrationality but uses the standard algebraic proof which is less novel than submission 1's approach. Rejecting to prevent redundancy with submission 1.", - "summary": "Redundant with co-submitted submission 1 which provides a more novel approach." + "reasoning": "Submission 2 proposes the same isolation boundary but omits the recovery and test plan already supplied by submission 1. Rejecting the weaker duplicate.", + "summary": "Redundant with co-submitted submission 1, which is stronger and more complete." }, { "submission_number": 3, "decision": "accept", - "reasoning": "Submission 3 explores continued fraction representations - completely different topic from submissions 1 and 2. Adds unique value.", + "reasoning": "Submission 3 provides a rigorous counterexample to a mathematical assumption used elsewhere and states the derivation explicitly. It adds distinct value.", "summary": "" } ] @@ -655,14 +632,14 @@ def get_validator_triple_json_schema() -> str: { "submission_number": 2, "decision": "reject", - "reasoning": "Submission 2 contains vague claims without mathematical rigor.", - "summary": "Too vague and lacks mathematical rigor." + "reasoning": "Submission 2 claims the design will improve reliability but specifies no mechanism, constraints, evidence, failure modes, or verification path.", + "summary": "Too vague to assess or implement; provide a concrete mechanism and verification plan." }, { "submission_number": 3, "decision": "reject", - "reasoning": "Submission 3 contains a logical fallacy in its central argument.", - "summary": "Contains logical fallacy - invalid proof structure." + "reasoning": "Submission 3 treats correlation as causal without identifying assumptions, confounders, or a falsifiable test.", + "summary": "Unsupported causal inference; state assumptions and provide a test that could falsify the claim." } ] } @@ -733,9 +710,9 @@ def get_cleanup_review_system_prompt() -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content -""" + EMPIRICAL_PROVENANCE_VALIDATION_RULES + """ +""" + EMPIRICAL_PROVENANCE_VALIDATION_RULES + "\n\n" + ALL_PURPOSE_VALIDATION_CRITERIA + "\n\n" + LEAN_VERIFIED_SUBMISSION_RULES + """ - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to evaluate rigorous, defensible, and verifiable content under the standards appropriate to its claims. Use internal context as exploration history, not as authority. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -756,14 +733,14 @@ def get_cleanup_review_system_prompt() -> str: 2. CONTRADICTS other accepted submissions (logical inconsistencies discovered) 3. Contains information that is now SUPERSEDED by better, more complete submissions 4. Was MARGINALLY useful initially but provides no unique value given the current database state -5. Contains claims that CONFLICT with established mathematical principles evident in other submissions +5. Contains claims that are incorrect or indefensible under the relevant domain and claim-type standards 6. Contains unsupported empirical or artifact claims presented as established fact REASONS TO KEEP - A submission should be kept if it: 1. Directly answers the user's whole problem or a necessary piece of it better than alternatives 2. Provides unique information that materially strengthens a direct route to the user's full prompt 3. Offers a different perspective or approach that materially improves the best direct solution path -4. Contains specific mathematical details, proofs, or techniques that are necessary for direct prompt progress +4. Contains a unique mechanism, design, algorithm, evidence plan, risk analysis, mathematical detail, proof, or technique necessary for direct prompt progress 5. Contributes to solution diversity only when that diversity improves credible direct-answer progress CONSERVATIVE APPROACH: @@ -811,14 +788,14 @@ def get_cleanup_review_json_schema() -> str: { "should_remove": false, "submission_number": null, - "reasoning": "All submissions contribute unique value. While submissions #3 and #7 both discuss transcendental numbers, #3 focuses on the Lindemann-Weierstrass theorem while #7 addresses continued fraction approximations - both are necessary." + "reasoning": "All submissions contribute unique value. Submission #3 defines the battery isolation mechanism and constraints, while #7 supplies a distinct fault-injection validation plan and failure criteria; neither is fully covered by the other." } Example (Removal Recommended): { "should_remove": true, "submission_number": 4, - "reasoning": "Submission #4 provides a basic definition of algebraic numbers which is now fully covered by submission #12's comprehensive treatment of algebraic vs transcendental classification. Submission #4 adds no unique information that isn't better explained in #12." + "reasoning": "Submission #4 gives a generic retry suggestion that is fully covered by submission #12's more complete backoff mechanism, constraints, failure handling, and verification plan. Submission #4 adds no unique value." } """ @@ -885,9 +862,9 @@ def get_removal_validation_system_prompt() -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content -""" + EMPIRICAL_PROVENANCE_VALIDATION_RULES + """ +""" + EMPIRICAL_PROVENANCE_VALIDATION_RULES + "\n\n" + ALL_PURPOSE_VALIDATION_CRITERIA + "\n\n" + LEAN_VERIFIED_SUBMISSION_RULES + """ - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to evaluate rigorous, defensible, and verifiable content under the standards appropriate to its claims. Use internal context as exploration history, not as authority. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -950,13 +927,13 @@ def get_removal_validation_json_schema() -> str: Example (Approve Removal): { "decision": "accept", - "reasoning": "The removal is justified. Submission #4's basic algebraic number definition is completely subsumed by submission #12's comprehensive classification. No unique information would be lost." + "reasoning": "The removal is justified. Submission #4's generic cache suggestion is completely subsumed by submission #12's concrete invalidation mechanism, constraints, failure modes, and validation plan. No unique information would be lost." } Example (Reject Removal): { "decision": "reject", - "reasoning": "While submission #4 overlaps with #12, it provides a simplified introductory explanation useful for foundational understanding. The database benefits from having both rigorous and accessible explanations." + "reasoning": "While submission #4 overlaps with #12, it uniquely identifies the sensor-drift failure mode and a calibration test that #12 does not cover. Removing it would discard objective-serving value." } """ diff --git a/backend/aggregator/validation/json_validator.py b/backend/aggregator/validation/json_validator.py index abfe0c7..92f9b64 100644 --- a/backend/aggregator/validation/json_validator.py +++ b/backend/aggregator/validation/json_validator.py @@ -498,7 +498,9 @@ def validate_compiler_validator_json(self, llm_output: str) -> Tuple[bool, Optio schema = { "decision": str, "reasoning": str, - "summary": str + "coherence_check": bool, + "rigor_check": bool, + "placement_check": bool, } valid, parsed, error = self.extract_and_validate_json(llm_output, schema) @@ -506,6 +508,8 @@ def validate_compiler_validator_json(self, llm_output: str) -> Tuple[bool, Optio # Additional validation for decision field if parsed["decision"] not in ["accept", "reject"]: return False, None, f"Invalid decision value: {parsed['decision']}. Must be 'accept' or 'reject'" + if not parsed["reasoning"].strip(): + return False, None, "reasoning must be a non-empty string" return valid, parsed, error diff --git a/backend/api/main.py b/backend/api/main.py index 89a2c4a..7dd3d4c 100644 --- a/backend/api/main.py +++ b/backend/api/main.py @@ -35,6 +35,7 @@ from backend.shared.config import rag_config, system_config from backend.shared.lean4_client import clear_lean4_client, close_lean4_client, initialize_lean4_client from backend.shared.runtime_settings import apply_persisted_runtime_settings +from backend.shared.workflow_start_guard import workflow_start_guard from backend.aggregator.core.coordinator import coordinator from backend.compiler.core.compiler_coordinator import compiler_coordinator from backend.autonomous.core.autonomous_coordinator import autonomous_coordinator @@ -168,10 +169,8 @@ def _restore_desktop_provider_credentials(api_client_manager) -> None: @asynccontextmanager -async def lifespan(app: FastAPI): +async def _application_lifespan(app: FastAPI): """Lifespan events for the FastAPI app.""" - _apply_generic_mode_from_env() - _validate_generic_mode_startup_env() _ensure_desktop_api_token() # Startup @@ -188,6 +187,9 @@ async def lifespan(app: FastAPI): Path(system_config.user_uploads_dir).mkdir(parents=True, exist_ok=True) apply_persisted_runtime_settings() + from backend.aggregator.core.rag_manager import rag_manager + await rag_manager.prepare_process_generation_cache() + from backend.shared.api_client_manager import api_client_manager if system_config.generic_mode: @@ -265,9 +267,21 @@ async def lifespan(app: FastAPI): # Restore saved LeanOJ state for the UI, but only launch model work when # explicitly requested. Lean 4 being enabled is not enough to imply that # LM Studio/OpenRouter models are loaded and ready at backend startup. - await leanoj_coordinator.restore_latest_session( - auto_resume=system_config.lean4_enabled and system_config.leanoj_auto_resume_enabled + auto_resume_leanoj = ( + system_config.lean4_enabled and system_config.leanoj_auto_resume_enabled ) + await leanoj_coordinator.restore_latest_session(auto_resume=False) + if auto_resume_leanoj and leanoj_coordinator._request is not None: + async with workflow_start_guard.reserve(): + if workflow_start_guard.active_owner is None: + from backend.api.routes.leanoj import ( + commit_leanoj_workflow_lease, + release_leanoj_workflow_lease, + ) + if leanoj_coordinator.start_in_background(): + commit_leanoj_workflow_lease() + if not leanoj_coordinator.is_active: + release_leanoj_workflow_lease() except Exception as exc: logger.warning("Failed to restore LeanOJ session state on startup: %s", exc) @@ -308,10 +322,23 @@ async def _warm_start_lean4() -> None: logger.debug("Lean 4 warm start task cancelled during shutdown") except Exception as exc: logger.debug("Lean 4 warm start task failed during shutdown: %s", exc) - await coordinator.stop() - await compiler_coordinator.stop() - await autonomous_coordinator.stop() - await leanoj_coordinator.stop() + try: + for label, stop in ( + ("Aggregator", coordinator.stop), + ("Compiler", compiler_coordinator.stop), + ("Autonomous Research", autonomous_coordinator.stop), + ("Proof Solver", leanoj_coordinator.stop), + ): + try: + await stop() + except Exception: + logger.exception("%s shutdown cleanup failed", label) + finally: + workflow_start_guard.release_all() + try: + await rag_manager.close() + except Exception: + logger.exception("Chroma shutdown cleanup failed") await close_lean4_client() clear_lean4_client() await lm_studio_client.close() @@ -322,6 +349,21 @@ async def _warm_start_lean4() -> None: logger.info("Shutdown complete") +@asynccontextmanager +async def lifespan(app: FastAPI): + """Hold authoritative data-root ownership for the complete app lifespan.""" + _apply_generic_mode_from_env() + _validate_generic_mode_startup_env() + from backend.shared.runtime_root_lock import RuntimeRootLease + + runtime_root_lease = RuntimeRootLease(system_config.data_dir).acquire() + try: + async with _application_lifespan(app): + yield + finally: + runtime_root_lease.release() + + # Create FastAPI app app = FastAPI( title="ASI Aggregator System", diff --git a/backend/api/routes/aggregator.py b/backend/api/routes/aggregator.py index 4946d2f..330c3c4 100644 --- a/backend/api/routes/aggregator.py +++ b/backend/api/routes/aggregator.py @@ -3,6 +3,7 @@ """ from fastapi import APIRouter, HTTPException, UploadFile, File from typing import List, Optional +import asyncio import logging from pathlib import Path import aiofiles @@ -16,7 +17,7 @@ from backend.shared.path_safety import resolve_path_within_root, validate_single_path_component from backend.shared.log_redaction import redact_log_text from backend.shared.manual_proof_context import get_manual_proof_context_lock -from backend.shared.workflow_start_guard import workflow_start_guard +from backend.shared.workflow_start_guard import WorkflowLease, workflow_start_guard from backend.shared.api_client_manager import api_client_manager from backend.shared.proof_search.assistant_coordinator import assistant_proof_search_coordinator from backend.aggregator.core.coordinator import coordinator @@ -24,8 +25,10 @@ from backend.aggregator.memory.event_log import event_log from backend.aggregator.memory.shared_training import ( clear_manual_aggregator_prompt, + clear_manual_main_submitter_config, load_manual_aggregator_prompt, save_manual_aggregator_prompt, + save_manual_main_submitter_config, shared_training_memory, ) from backend.autonomous.core.proof_verification_stage import ProofVerificationStage @@ -39,6 +42,18 @@ router = APIRouter(prefix="/api/aggregator", tags=["aggregator"]) MAX_UPLOAD_BYTES = 5 * 1024 * 1024 +ALLOWED_UPLOAD_EXTENSIONS = {".txt", ".lean"} +AGGREGATOR_WORKFLOW_OWNER = "manual_aggregator" +_aggregator_workflow_lease: WorkflowLease | None = None + + +def _release_aggregator_workflow_lease() -> None: + global _aggregator_workflow_lease + workflow_start_guard.release(_aggregator_workflow_lease) + _aggregator_workflow_lease = None + + +coordinator.top_level_terminal_callback = _release_aggregator_workflow_lease MANUAL_PROOF_ACTIVE_KEYS = { "brainstorm:manual_aggregator", @@ -46,6 +61,40 @@ } +async def _delete_uploaded_file(file_ref: str) -> bool: + """Delete a logical upload filename from the upload root.""" + safe_filename = validate_single_path_component(file_ref, "filename") + if Path(safe_filename).suffix.lower() not in ALLOWED_UPLOAD_EXTENSIONS: + raise ValueError("Only .txt and .lean uploads are managed here") + + uploads_dir = Path(system_config.user_uploads_dir) + file_path = resolve_path_within_root(uploads_dir, safe_filename) + if not file_path.exists(): + return False + if not file_path.is_file(): + raise ValueError("Upload path is not a file") + + await asyncio.to_thread(file_path.unlink) + return True + + +async def _clear_uploaded_files() -> int: + """Clear text/Lean user uploads so stale files cannot seed later workflows.""" + uploads_dir = Path(system_config.user_uploads_dir) + if not uploads_dir.exists(): + return 0 + + deleted = 0 + for file_path in uploads_dir.iterdir(): + if ( + file_path.is_file() + and file_path.suffix.lower() in ALLOWED_UPLOAD_EXTENSIONS + ): + await asyncio.to_thread(file_path.unlink) + deleted += 1 + return deleted + + async def _manual_proof_clear_blocker() -> Optional[str]: """Return a blocker message if manual proof work could write stale proofs.""" active_keys = await ProofVerificationStage.active_source_keys() @@ -145,6 +194,10 @@ async def _require_openrouter_host_provider_available( def _get_start_conflict() -> Optional[str]: """Return a user-facing conflict message if another workflow is active.""" + if workflow_start_guard.active_owner: + if workflow_start_guard.active_owner == AGGREGATOR_WORKFLOW_OWNER: + return "Aggregator is already running" + return "Cannot start Aggregator while another workflow is running. Stop it first." if coordinator.is_running: return "Aggregator is already running" @@ -208,6 +261,10 @@ async def _ensure_manual_event_log_loaded_for_read() -> None: @router.post("/start") async def start_aggregator(request: AggregatorStartRequest): """Start the aggregator system.""" + global _aggregator_workflow_lease + manual_solution_path = None + parent_start_committed = False + coordinator_started = False try: async with workflow_start_guard.reserve(): conflict = _get_start_conflict() @@ -349,6 +406,106 @@ async def start_aggregator(request: AggregatorStartRequest): ), ) + # One durable plan spans the active manual Aggregator -> Compiler run. + from pathlib import Path + from backend.shared.solution_path import ( + build_review_prompt, + compact_review_prompt, + review_with_json_retry, + solution_path_registry, + ) + primary_submitter = next( + ( + config + for config in request.submitter_configs + if config.submitter_id == 1 + ), + None, + ) + if primary_submitter is None: + raise ValueError( + "Main Submitter 1 is required for solution-path review." + ) + await save_manual_main_submitter_config( + primary_submitter.model_dump(mode="json") + ) + reviewer_role_id = "manual_solution_path_reviewer" + api_client_manager.configure_role( + reviewer_role_id, + ModelConfig( + provider=primary_submitter.provider, + model_id=primary_submitter.model_id, + openrouter_model_id=( + primary_submitter.model_id + if primary_submitter.provider == "openrouter" + else None + ), + openrouter_provider=primary_submitter.openrouter_provider, + openrouter_reasoning_effort=primary_submitter.openrouter_reasoning_effort, + lm_studio_fallback_id=primary_submitter.lm_studio_fallback_id, + context_window=primary_submitter.context_window, + max_output_tokens=primary_submitter.max_output_tokens, + supercharge_enabled=primary_submitter.supercharge_enabled, + ), + ) + + async def review_solution_path(proposal, current_plan): + prompt = build_review_prompt( + user_prompt=request.user_prompt, + proposal=proposal, + current_plan=current_plan, + ) + from backend.shared.response_extraction import extract_message_text + + async def call(messages): + return await api_client_manager.generate_completion( + task_id=f"agg_sub1_solution_path_{proposal.review_count:03d}", + role_id=reviewer_role_id, + model=primary_submitter.model_id, + messages=messages, + temperature=0.0, + max_tokens=primary_submitter.max_output_tokens, + ) + + return await review_with_json_retry( + prompt=prompt, + call_completion=call, + extract_text=lambda response: extract_message_text( + response["choices"][0]["message"] + ), + context_window=primary_submitter.context_window, + max_output_tokens=primary_submitter.max_output_tokens, + compact_prompt=compact_review_prompt( + user_prompt=request.user_prompt, + proposal=proposal, + current_plan=current_plan, + ), + ) + + manual_solution_path = await solution_path_registry.acquire( + Path(system_config.data_dir) / "solution_paths", + workflow_mode="manual", + user_prompt=request.user_prompt, + stable_run_id="manual", + reviewer=review_solution_path, + ) + # Keep Assistant as the final configured role for compatibility + # with settings/defaulting observers; the dedicated reviewer keeps + # its independent role ID and immutable copied configuration. + api_client_manager.configure_role( + "aggregator_assistant", + ModelConfig( + provider=assistant_provider, + model_id=assistant_model, + openrouter_provider=assistant_openrouter_provider, + openrouter_reasoning_effort=assistant_reasoning_effort, + lm_studio_fallback_id=assistant_fallback, + context_window=assistant_context_size, + max_output_tokens=assistant_max_output_tokens, + supercharge_enabled=assistant_supercharge_enabled, + ), + ) + # Initialize coordinator with per-submitter configs (includes OpenRouter provider fields) await coordinator.initialize( user_prompt=request.user_prompt, @@ -364,11 +521,19 @@ async def start_aggregator(request: AggregatorStartRequest): validator_lm_studio_fallback=request.validator_lm_studio_fallback, validator_supercharge_enabled=request.validator_supercharge_enabled, creativity_emphasis_boost_enabled=request.creativity_emphasis_boost_enabled, + solution_path_manager=manual_solution_path, ) # Start coordinator token_tracker.reset() token_tracker.start_timer() await coordinator.start() + coordinator_started = coordinator.is_running + if not coordinator_started: + raise RuntimeError("Aggregator did not enter running state") + _aggregator_workflow_lease = workflow_start_guard.commit( + AGGREGATOR_WORKFLOW_OWNER + ) + parent_start_committed = True return { "status": "started", @@ -386,22 +551,34 @@ async def start_aggregator(request: AggregatorStartRequest): # Other errors logger.error(f"Failed to start aggregator: {e}", exc_info=True) raise HTTPException(status_code=500, detail="Internal server error") + finally: + if coordinator_started and not parent_start_committed: + await coordinator.stop() + token_tracker.stop_timer() + if manual_solution_path is not None and not parent_start_committed: + await manual_solution_path.stop() @router.post("/stop") async def stop_aggregator(): """Stop the aggregator system.""" - try: - await coordinator.stop() - await assistant_proof_search_coordinator.stop_all( - broadcast=True, - reason="aggregator_stopped", - ) - token_tracker.stop_timer() - return {"status": "stopped", "message": "Aggregator system stopped"} - except Exception as e: - logger.error(f"Failed to stop aggregator: {e}") - raise HTTPException(status_code=500, detail="Internal server error") + async with workflow_start_guard.reserve(): + try: + await coordinator.stop() + if getattr(coordinator, "solution_path_manager", None) is not None: + await coordinator.solution_path_manager.stop() + await assistant_proof_search_coordinator.stop_all( + broadcast=True, + reason="aggregator_stopped", + ) + if coordinator.is_running: + raise RuntimeError("Aggregator remained active after stop") + token_tracker.stop_timer() + _release_aggregator_workflow_lease() + return {"status": "stopped", "message": "Aggregator system stopped"} + except Exception as e: + logger.error(f"Failed to stop aggregator: {e}") + raise HTTPException(status_code=500, detail="Internal server error") @router.get("/status", response_model=SystemStatus) @@ -467,7 +644,7 @@ async def save_results(): async def clear_all_submissions(): """Clear all accepted submissions and reset the system.""" try: - async with get_manual_proof_context_lock(): + async with workflow_start_guard.reserve(), get_manual_proof_context_lock(): if compiler_coordinator.is_running: raise HTTPException( status_code=409, @@ -478,23 +655,32 @@ async def clear_all_submissions(): raise HTTPException(status_code=409, detail=blocker) if coordinator.is_running: await coordinator.stop() + _release_aggregator_workflow_lease() archived_proofs = await manual_proof_database.archive_current_run( Path(system_config.data_dir) / "manual_proof_runs", user_prompt=await load_manual_aggregator_prompt(), reason="manual_aggregator_clear_all", ) await coordinator.clear_all_submissions() + from backend.shared.solution_path import solution_path_registry + await solution_path_registry.clear_run( + Path(system_config.data_dir) / "solution_paths", "manual" + ) + coordinator.solution_path_manager = None await assistant_proof_search_coordinator.stop_all( broadcast=True, reason="aggregator_cleared", ) await assistant_proof_search_coordinator.clear_cooldown_state() await clear_manual_aggregator_prompt() + await clear_manual_main_submitter_config() + deleted_uploads = await _clear_uploaded_files() return { "status": "cleared", "message": "All submissions cleared and system reset", "archived_manual_proofs": archived_proofs, + "deleted_uploads": deleted_uploads, } except HTTPException: raise @@ -508,12 +694,18 @@ async def upload_file(file: UploadFile = File(...)): """Upload a user file.""" try: safe_filename = validate_single_path_component(file.filename, "filename") - if not safe_filename.lower().endswith(".txt"): - raise HTTPException(status_code=400, detail="Only .txt uploads are supported") + if Path(safe_filename).suffix.lower() not in ALLOWED_UPLOAD_EXTENSIONS: + raise HTTPException(status_code=400, detail="Only .txt and .lean uploads are supported") content = await file.read(MAX_UPLOAD_BYTES + 1) if len(content) > MAX_UPLOAD_BYTES: raise HTTPException(status_code=413, detail="Upload exceeds 5 MB limit") + try: + decoded_content = content.decode("utf-8") + except UnicodeDecodeError: + raise HTTPException(status_code=400, detail="Uploads must be UTF-8 encoded text files") + if not decoded_content.strip(): + raise HTTPException(status_code=400, detail="Upload is empty or contains only whitespace") uploads_dir = Path(system_config.user_uploads_dir) uploads_dir.mkdir(parents=True, exist_ok=True) @@ -537,6 +729,24 @@ async def upload_file(file: UploadFile = File(...)): raise HTTPException(status_code=500, detail="Internal server error") +@router.delete("/upload-file/{filename}") +async def delete_uploaded_file(filename: str): + """Remove a previously uploaded user file.""" + try: + deleted = await _delete_uploaded_file(filename) + return { + "status": "deleted" if deleted else "not_found", + "filename": validate_single_path_component(filename, "filename"), + "deleted": deleted, + } + except ValueError as e: + logger.warning("Rejected unsafe upload deletion request: %s", e) + raise HTTPException(status_code=400, detail=str(e)) + except Exception as e: + logger.error(f"Failed to delete uploaded file: {e}") + raise HTTPException(status_code=500, detail="Internal server error") + + @router.get("/models", response_model=List[ModelInfo]) async def get_models(): """Get available models from LM Studio.""" diff --git a/backend/api/routes/autonomous.py b/backend/api/routes/autonomous.py index 29c0721..26a7613 100644 --- a/backend/api/routes/autonomous.py +++ b/backend/api/routes/autonomous.py @@ -29,13 +29,26 @@ from backend.shared.config import system_config from backend.shared.embedding_readiness import require_embedding_provider_ready from backend.shared.log_redaction import redact_log_text -from backend.shared.workflow_start_guard import workflow_start_guard +from backend.shared.workflow_start_guard import WorkflowLease, workflow_start_guard from backend.shared.response_extraction import extract_message_text from backend.shared.proof_search.assistant_coordinator import assistant_proof_search_coordinator logger = logging.getLogger(__name__) router = APIRouter(prefix="/api/auto-research", tags=["autonomous"]) +AUTONOMOUS_WORKFLOW_OWNER = "autonomous" +_autonomous_workflow_lease: WorkflowLease | None = None + + +def _release_autonomous_workflow_lease() -> None: + global _autonomous_workflow_lease + workflow_start_guard.release(_autonomous_workflow_lease) + _autonomous_workflow_lease = None + + +autonomous_coordinator.top_level_terminal_callback = ( + _release_autonomous_workflow_lease +) def _get_active_autonomous_session_id() -> str: @@ -309,6 +322,10 @@ async def _get_combined_api_logs( def _get_start_conflict() -> Optional[str]: """Return a user-facing conflict message if another workflow is active.""" + if workflow_start_guard.active_owner: + if workflow_start_guard.active_owner == AUTONOMOUS_WORKFLOW_OWNER: + return "Autonomous research is already running" + return "Cannot start Autonomous Research while another workflow is running. Stop it first." autonomous_state = autonomous_coordinator.get_state() if autonomous_state.is_running or autonomous_coordinator.is_active: return "Autonomous research is already running" @@ -714,6 +731,7 @@ async def _delete_autonomous_paper_from_scope( @router.post("/start") async def start_autonomous_research(request: AutonomousResearchStartRequest): """Start autonomous research mode.""" + global _autonomous_workflow_lease try: from backend.shared.config import system_config @@ -847,8 +865,19 @@ async def start_autonomous_research(request: AutonomousResearchStartRequest): ) # Start in background with a retained task handle so Stop can cancel it. - if not autonomous_coordinator.start_in_background(): - raise HTTPException(status_code=400, detail="Autonomous research is already running") + try: + _autonomous_workflow_lease = workflow_start_guard.commit( + AUTONOMOUS_WORKFLOW_OWNER + ) + if not autonomous_coordinator.start_in_background(): + raise HTTPException(status_code=400, detail="Autonomous research is already running") + except BaseException: + if autonomous_coordinator.is_active: + await autonomous_coordinator.stop() + _release_autonomous_workflow_lease() + raise + if not autonomous_coordinator.is_active: + _release_autonomous_workflow_lease() return { "success": True, @@ -871,36 +900,55 @@ async def start_autonomous_research(request: AutonomousResearchStartRequest): @router.post("/stop") async def stop_autonomous_research(): """Stop autonomous research mode gracefully.""" - try: - state = autonomous_coordinator.get_state() - if not state.is_running and not autonomous_coordinator.is_active: + # Start performs substantial coordinator and RAG initialization while holding + # this process-wide lifecycle reservation. A Stop that overlaps that work can + # otherwise tear down coordinator state while Chroma is still being reset. + async with workflow_start_guard.reserve(): + stopped_cleanly = False + try: + state = autonomous_coordinator.get_state() + if not state.is_running and not autonomous_coordinator.is_active: + return { + "success": True, + "message": "Autonomous research was not running" + } + + await autonomous_coordinator.stop() + await assistant_proof_search_coordinator.stop_all( + broadcast=True, + reason="autonomous_stopped", + ) + + # Get final stats + stats = await research_metadata.get_stats() + stopped_cleanly = not autonomous_coordinator.is_active + return { "success": True, - "message": "Autonomous research was not running" + "message": "Autonomous research stopped", + "final_stats": stats } - - await autonomous_coordinator.stop() - await assistant_proof_search_coordinator.stop_all( - broadcast=True, - reason="autonomous_stopped", - ) - - # Get final stats - stats = await research_metadata.get_stats() - - return { - "success": True, - "message": "Autonomous research stopped", - "final_stats": stats - } - - except Exception as e: - logger.error(f"Failed to stop autonomous research: {e}") - raise HTTPException(status_code=500, detail="Internal server error") + + except Exception as e: + logger.error(f"Failed to stop autonomous research: {e}") + raise HTTPException(status_code=500, detail="Internal server error") + finally: + if stopped_cleanly or not autonomous_coordinator.is_active: + _release_autonomous_workflow_lease() + else: + logger.error( + "Autonomous stop did not reach a terminal state; retaining workflow ownership" + ) @router.post("/clear") async def clear_autonomous_research(confirm: bool = False): + """Serialize clear against pending top-level lifecycle transitions.""" + async with workflow_start_guard.reserve(): + return await _clear_autonomous_research_impl(confirm) + + +async def _clear_autonomous_research_impl(confirm: bool = False): """Clear all autonomous research data. Returns success even with non-critical warnings. @@ -913,11 +961,10 @@ async def clear_autonomous_research(confirm: bool = False): detail="Must confirm with confirm=true" ) - state = autonomous_coordinator.get_state() - if state.is_running: + if autonomous_coordinator.is_active: raise HTTPException( status_code=400, - detail="Cannot clear data while autonomous research is running. Please stop research first." + detail="Cannot clear data while autonomous research is active or still stopping. Please wait for Stop to finish." ) logger.info("Starting autonomous research data clear...") @@ -3057,8 +3104,14 @@ async def get_autonomous_api_log_detail(log_key: str, workflow: Optional[str] = "log": { **log, "log_key": log_key, - "prompt_size": len(str(log.get("prompt_full") or "")), - "response_size": len(str(log.get("response_full") or "")), + "prompt_size": int( + log.get("prompt_size") + or len(str(log.get("prompt_full") or "")) + ), + "response_size": int( + log.get("response_size") + or len(str(log.get("response_full") or "")) + ), }, } diff --git a/backend/api/routes/compiler.py b/backend/api/routes/compiler.py index 4842b98..0ad8b01 100644 --- a/backend/api/routes/compiler.py +++ b/backend/api/routes/compiler.py @@ -17,7 +17,7 @@ from backend.shared.api_client_manager import api_client_manager from backend.shared.log_redaction import redact_log_text from backend.shared.manual_proof_context import get_manual_proof_context_lock -from backend.shared.workflow_start_guard import workflow_start_guard +from backend.shared.workflow_start_guard import WorkflowLease, workflow_start_guard from backend.shared.proof_search.assistant_coordinator import assistant_proof_search_coordinator from backend.compiler.core.compiler_coordinator import CRITIQUE_ATTEMPT_TARGET, compiler_coordinator from backend.compiler.memory.manual_prompt import ( @@ -31,6 +31,8 @@ from backend.aggregator.memory.shared_training import ( append_proof_to_manual_shared_training, clear_manual_shared_training_proof_appendix, + load_manual_aggregator_prompt, + load_manual_main_submitter_config, ) from backend.autonomous.core.autonomous_coordinator import autonomous_coordinator from backend.autonomous.core.proof_verification_stage import ProofVerificationStage @@ -46,11 +48,29 @@ _compiler_proof_only_task: asyncio.Task | None = None _saved_compiler_proof_tasks: set[asyncio.Task] = set() MANUAL_AGGREGATOR_SOURCE_ID = "manual_aggregator" +COMPILER_WORKFLOW_OWNER = "manual_compiler" +COMPILER_PROOF_ONLY_OWNER = "manual_compiler_proof_only" +_compiler_workflow_lease: WorkflowLease | None = None +_compiler_proof_only_lease: WorkflowLease | None = None MANUAL_PROOF_ACTIVE_KEYS = { "brainstorm:manual_aggregator", "paper:manual_compiler_current", } +def _release_compiler_workflow_lease() -> None: + global _compiler_workflow_lease + workflow_start_guard.release(_compiler_workflow_lease) + _compiler_workflow_lease = None + + +def _release_compiler_proof_only_lease() -> None: + global _compiler_proof_only_lease + workflow_start_guard.release(_compiler_proof_only_lease) + _compiler_proof_only_lease = None + + +compiler_coordinator.top_level_terminal_callback = _release_compiler_workflow_lease + async def _release_pre_reserved_source(source_type: str, source_id: str, reserved: bool) -> None: if reserved and source_id: @@ -218,6 +238,12 @@ async def _run_saved_compiler_paper_proof_check( def _get_start_conflict() -> str | None: """Return a user-facing conflict message if another workflow is active.""" + if workflow_start_guard.active_owner: + if workflow_start_guard.active_owner == COMPILER_PROOF_ONLY_OWNER: + return "Compiler proof verification is already running" + if workflow_start_guard.active_owner == COMPILER_WORKFLOW_OWNER: + return "Compiler is already running" + return "Cannot start Compiler while another workflow is running. Stop it first." if compiler_coordinator.is_running: return "Compiler is already running" @@ -356,6 +382,21 @@ async def _run_compiler_aggregator_proof_check( token_tracker.stop_timer() +async def _run_owned_compiler_aggregator_proof_check( + request: CompilerStartRequest, + *, + source_reserved: bool = False, +) -> None: + """Keep proof-only workflow ownership tied to the exact background task.""" + try: + await _run_compiler_aggregator_proof_check( + request, + source_reserved=source_reserved, + ) + finally: + _release_compiler_proof_only_lease() + + def _log_background_task_failure(task: asyncio.Task) -> None: _saved_compiler_proof_tasks.discard(task) try: @@ -386,7 +427,10 @@ async def _manual_proof_clear_blocker() -> str | None: @router.post("/start") async def start_compiler(request: CompilerStartRequest): """Start the compiler system.""" - global _compiler_proof_only_task + global _compiler_proof_only_task, _compiler_workflow_lease, _compiler_proof_only_lease + manual_solution_path = None + parent_start_committed = False + coordinator_started = False try: async with workflow_start_guard.reserve(): conflict = _get_start_conflict() @@ -449,10 +493,29 @@ async def start_compiler(request: CompilerStartRequest): await ProofVerificationStage.reserve_source("brainstorm", MANUAL_AGGREGATOR_SOURCE_ID) except RuntimeError: raise HTTPException(status_code=409, detail="A proof verification is already running for the manual Aggregator database.") - _compiler_proof_only_task = asyncio.create_task( - _run_compiler_aggregator_proof_check(request, source_reserved=True) - ) - _compiler_proof_only_task.add_done_callback(_log_background_task_failure) + try: + _compiler_proof_only_lease = workflow_start_guard.commit( + COMPILER_PROOF_ONLY_OWNER + ) + _compiler_proof_only_task = asyncio.create_task( + _run_owned_compiler_aggregator_proof_check( + request, + source_reserved=True, + ) + ) + _compiler_proof_only_task.add_done_callback(_log_background_task_failure) + except BaseException: + task_was_started = _compiler_proof_only_task is not None + if _compiler_proof_only_task is not None: + _compiler_proof_only_task.cancel() + await asyncio.gather(_compiler_proof_only_task, return_exceptions=True) + _compiler_proof_only_task = None + _release_compiler_proof_only_lease() + if not task_was_started: + await ProofVerificationStage.release_source( + "brainstorm", MANUAL_AGGREGATOR_SOURCE_ID + ) + raise return { "status": "proof_check_started", "message": "Compiler proof verification started over the Aggregator database", @@ -520,6 +583,70 @@ async def start_compiler(request: CompilerStartRequest): redact_log_text(request.high_param_max_output_tokens, 40), ) + # Continue the durable manual Aggregator plan, including after a + # backend restart where the process-local registry is empty. + from backend.shared.solution_path import ( + build_review_prompt, + compact_review_prompt, + review_with_json_retry, + solution_path_registry, + ) + manual_prompt = (await load_manual_aggregator_prompt()).strip() + manual_solution_path = solution_path_registry.get("manual") + if manual_solution_path is None and manual_prompt: + saved_primary = await load_manual_main_submitter_config() + live_primary = api_client_manager.get_role_config("aggregator_submitter_1") + primary = live_primary or ( + ModelConfig(**saved_primary) if saved_primary else None + ) + if primary is not None: + api_client_manager.configure_role("aggregator_submitter_1", primary) + reviewer_role_id = "manual_solution_path_reviewer" + api_client_manager.configure_role(reviewer_role_id, primary) + + async def review_solution_path(proposal, current_plan): + prompt = build_review_prompt( + user_prompt=manual_prompt, + proposal=proposal, + current_plan=current_plan, + ) + + async def call(messages): + return await api_client_manager.generate_completion( + task_id=f"agg_sub1_solution_path_{proposal.review_count:03d}", + role_id=reviewer_role_id, + model=primary.model_id, + messages=messages, + temperature=0.0, + max_tokens=primary.max_output_tokens, + ) + + return await review_with_json_retry( + prompt=prompt, + call_completion=call, + extract_text=lambda response: extract_message_text( + response["choices"][0]["message"] + ), + context_window=primary.context_window, + max_output_tokens=primary.max_output_tokens, + compact_prompt=compact_review_prompt( + user_prompt=manual_prompt, + proposal=proposal, + current_plan=current_plan, + ), + ) + + manual_solution_path = await solution_path_registry.acquire( + Path(system_config.data_dir) / "solution_paths", + workflow_mode="manual", + user_prompt=manual_prompt, + stable_run_id="manual", + reviewer=review_solution_path, + ) + elif manual_solution_path is not None: + await manual_solution_path.start() + compiler_coordinator.solution_path_manager = manual_solution_path + # Initialize coordinator with OpenRouter provider configurations await compiler_coordinator.initialize( compiler_prompt=request.compiler_prompt, @@ -555,6 +682,13 @@ async def start_compiler(request: CompilerStartRequest): token_tracker.reset() token_tracker.start_timer() await compiler_coordinator.start() + coordinator_started = compiler_coordinator.is_running + if not coordinator_started: + raise RuntimeError("Compiler did not enter running state") + _compiler_workflow_lease = workflow_start_guard.commit( + COMPILER_WORKFLOW_OWNER + ) + parent_start_committed = True return {"status": "started", "message": "Compiler started successfully"} @@ -607,27 +741,42 @@ async def start_compiler(request: CompilerStartRequest): # Other errors logger.error(f"Failed to start compiler: {e}", exc_info=True) raise HTTPException(status_code=500, detail="Internal server error") + finally: + if coordinator_started and not parent_start_committed: + await compiler_coordinator.stop() + token_tracker.stop_timer() + if manual_solution_path is not None and not parent_start_committed: + await manual_solution_path.stop() @router.post("/stop") async def stop_compiler(): """Stop the compiler system.""" global _compiler_proof_only_task - try: - if _compiler_proof_only_task and not _compiler_proof_only_task.done(): - _compiler_proof_only_task.cancel() - await asyncio.gather(_compiler_proof_only_task, return_exceptions=True) - _compiler_proof_only_task = None - await compiler_coordinator.stop() - await assistant_proof_search_coordinator.stop_all( - broadcast=True, - reason="compiler_stopped", - ) - token_tracker.stop_timer() - return {"status": "stopped", "message": "Compiler stopped"} - except Exception as e: - logger.error(f"Failed to stop compiler: {e}") - raise HTTPException(status_code=500, detail="Internal server error") + async with workflow_start_guard.reserve(): + try: + if _compiler_proof_only_task and not _compiler_proof_only_task.done(): + _compiler_proof_only_task.cancel() + await asyncio.gather(_compiler_proof_only_task, return_exceptions=True) + _compiler_proof_only_task = None + await compiler_coordinator.stop() + if getattr(compiler_coordinator, "solution_path_manager", None) is not None: + await compiler_coordinator.solution_path_manager.stop() + await assistant_proof_search_coordinator.stop_all( + broadcast=True, + reason="compiler_stopped", + ) + if compiler_coordinator.is_running or ( + _compiler_proof_only_task and not _compiler_proof_only_task.done() + ): + raise RuntimeError("Compiler remained active after stop") + token_tracker.stop_timer() + _release_compiler_proof_only_lease() + _release_compiler_workflow_lease() + return {"status": "stopped", "message": "Compiler stopped"} + except Exception as e: + logger.error(f"Failed to stop compiler: {e}") + raise HTTPException(status_code=500, detail="Internal server error") @router.post("/test-models") @@ -964,12 +1113,13 @@ async def clear_paper(confirm: bool = False): ) try: - async with get_manual_proof_context_lock(): + async with workflow_start_guard.reserve(), get_manual_proof_context_lock(): blocker = await _manual_proof_clear_blocker() if blocker: raise HTTPException(status_code=409, detail=blocker) if compiler_coordinator.is_running: await compiler_coordinator.stop() + _release_compiler_workflow_lease() persisted_prompt = compiler_coordinator.user_prompt or await load_manual_compiler_prompt() archived_proofs = await manual_proof_database.archive_current_run( Path(system_config.data_dir) / "manual_proof_runs", @@ -978,6 +1128,11 @@ async def clear_paper(confirm: bool = False): ) await clear_manual_shared_training_proof_appendix() await compiler_coordinator.clear_paper() + from backend.shared.solution_path import solution_path_registry + await solution_path_registry.clear_run( + Path(system_config.data_dir) / "solution_paths", "manual" + ) + compiler_coordinator.solution_path_manager = None await assistant_proof_search_coordinator.stop_all( broadcast=True, reason="compiler_cleared", diff --git a/backend/api/routes/connectivity.py b/backend/api/routes/connectivity.py index b750744..b94ae6e 100644 --- a/backend/api/routes/connectivity.py +++ b/backend/api/routes/connectivity.py @@ -309,6 +309,7 @@ async def update_connectivity_toggles(request: ConnectivityToggleRequest) -> dic broadcast=True, reason="agent_conversation_memory_disabled", ) + await assistant_proof_search_coordinator.clear_cooldown_state() if request.wolfram_alpha_enabled is not None: system_config.wolfram_alpha_enabled = bool(request.wolfram_alpha_enabled) diff --git a/backend/api/routes/leanoj.py b/backend/api/routes/leanoj.py index 7372d95..8eed561 100644 --- a/backend/api/routes/leanoj.py +++ b/backend/api/routes/leanoj.py @@ -16,11 +16,28 @@ from backend.shared.embedding_readiness import require_embedding_provider_ready from backend.shared.models import LeanOJStartRequest from backend.shared.proof_search.assistant_coordinator import assistant_proof_search_coordinator -from backend.shared.workflow_start_guard import workflow_start_guard +from backend.shared.workflow_start_guard import WorkflowLease, workflow_start_guard logger = logging.getLogger(__name__) router = APIRouter(prefix="/api/leanoj", tags=["leanoj"]) +LEANOJ_WORKFLOW_OWNER = "leanoj" +_leanoj_workflow_lease: WorkflowLease | None = None + + +def release_leanoj_workflow_lease() -> None: + global _leanoj_workflow_lease + workflow_start_guard.release(_leanoj_workflow_lease) + _leanoj_workflow_lease = None + + +def commit_leanoj_workflow_lease() -> WorkflowLease: + global _leanoj_workflow_lease + _leanoj_workflow_lease = workflow_start_guard.commit(LEANOJ_WORKFLOW_OWNER) + return _leanoj_workflow_lease + + +leanoj_coordinator.top_level_terminal_callback = release_leanoj_workflow_lease def _leanoj_sessions_base_dir() -> Path: @@ -66,6 +83,8 @@ def _leanoj_prompt(payload: dict[str, Any]) -> str: request_payload = _leanoj_request_payload(payload) return ( str(request_payload.get("user_prompt") or "").strip() + or str(payload.get("user_prompt") or "").strip() + or str(payload.get("prompt") or "").strip() or str(payload.get("selected_topic") or "").strip() or "Proof Solver problem" ) @@ -98,6 +117,7 @@ def _build_leanoj_final_proof(payload: dict[str, Any]) -> dict[str, Any] | None: "proof_id": proof_id, "shared_proof_id": shared_proof_id, "session_id": session_id, + "run_id": session_id, "proof_kind": "final", "theorem_name": "Final Proof Solver Submission", "theorem_statement": prompt, @@ -146,6 +166,7 @@ def _build_leanoj_subproofs(payload: dict[str, Any]) -> list[dict[str, Any]]: "proof_id": proof_id, "shared_proof_id": shared_proof_id, "session_id": session_id, + "run_id": session_id, "proof_kind": "subproof", "theorem_name": return_title, "theorem_statement": theorem_or_lemma or request_text or "Verified Proof Solver subproof", @@ -188,6 +209,7 @@ def _build_leanoj_session_summary(payload: dict[str, Any], proofs: list[dict[str subproof_count = sum(1 for proof in proofs if proof.get("proof_kind") == "subproof") return { "session_id": session_id, + "run_id": session_id, "user_prompt": prompt, "selected_topic": str(payload.get("selected_topic") or ""), "created_at": _leanoj_created_at(payload), @@ -209,6 +231,10 @@ def _sort_leanoj_proofs(proofs: list[dict[str, Any]]) -> list[dict[str, Any]]: def _get_start_conflict() -> Optional[str]: + if workflow_start_guard.active_owner: + if workflow_start_guard.active_owner == LEANOJ_WORKFLOW_OWNER: + return "Proof Solver is already running" + return "Cannot start Proof Solver while another workflow is running. Stop it first." if leanoj_coordinator.is_active: return "Proof Solver is already running" if coordinator.is_running: @@ -268,8 +294,17 @@ async def start_leanoj(request: LeanOJStartRequest): _validate_start_role_limits(request) await require_embedding_provider_ready() resumed = await leanoj_coordinator.resume_or_initialize(request) - if not leanoj_coordinator.start_in_background(): - raise HTTPException(status_code=400, detail="Proof Solver is already running") + try: + commit_leanoj_workflow_lease() + if not leanoj_coordinator.start_in_background(): + raise HTTPException(status_code=400, detail="Proof Solver is already running") + except BaseException: + if leanoj_coordinator.is_active: + await leanoj_coordinator.stop() + release_leanoj_workflow_lease() + raise + if not leanoj_coordinator.is_active: + release_leanoj_workflow_lease() return { "success": True, "message": "Proof Solver resumed" if resumed else "Proof Solver started", @@ -288,20 +323,24 @@ async def start_leanoj(request: LeanOJStartRequest): @router.post("/stop") async def stop_leanoj(): """Stop the active Proof Solver run.""" - try: - await leanoj_coordinator.stop() - await assistant_proof_search_coordinator.stop_all( - broadcast=True, - reason="leanoj_stopped", - ) - return { - "success": True, - "message": "Proof Solver stopped", - "status": leanoj_coordinator.get_status(), - } - except Exception as exc: - logger.exception("Failed to stop Proof Solver") - raise HTTPException(status_code=500, detail=str(exc)) + async with workflow_start_guard.reserve(): + try: + await leanoj_coordinator.stop() + await assistant_proof_search_coordinator.stop_all( + broadcast=True, + reason="leanoj_stopped", + ) + if leanoj_coordinator.is_active: + raise RuntimeError("Proof Solver remained active after stop") + release_leanoj_workflow_lease() + return { + "success": True, + "message": "Proof Solver stopped", + "status": leanoj_coordinator.get_status(), + } + except Exception as exc: + logger.exception("Failed to stop Proof Solver") + raise HTTPException(status_code=500, detail=str(exc)) @router.post("/clear") @@ -309,21 +348,23 @@ async def clear_leanoj(confirm: bool = False): """Clear saved Proof Solver progress.""" if not confirm: raise HTTPException(status_code=400, detail="Confirmation required. Use ?confirm=true to clear Proof Solver progress.") - try: - await leanoj_coordinator.clear() - await assistant_proof_search_coordinator.stop_all( - broadcast=True, - reason="leanoj_cleared", - ) - await assistant_proof_search_coordinator.clear_cooldown_state() - return { - "success": True, - "message": "Proof Solver progress cleared", - "status": leanoj_coordinator.get_status(), - } - except Exception as exc: - logger.exception("Failed to clear Proof Solver progress") - raise HTTPException(status_code=500, detail=str(exc)) + async with workflow_start_guard.reserve(): + try: + await leanoj_coordinator.clear() + await assistant_proof_search_coordinator.stop_all( + broadcast=True, + reason="leanoj_cleared", + ) + await assistant_proof_search_coordinator.clear_cooldown_state() + release_leanoj_workflow_lease() + return { + "success": True, + "message": "Proof Solver progress cleared", + "status": leanoj_coordinator.get_status(), + } + except Exception as exc: + logger.exception("Failed to clear Proof Solver progress") + raise HTTPException(status_code=500, detail=str(exc)) @router.get("/status") @@ -372,7 +413,7 @@ async def get_leanoj_library(include_subproofs: bool = True): current_status = leanoj_coordinator.get_status() current_session_id = str(current_status.get("session_id") or "") if current_session_id: - payloads_by_session[current_session_id] = current_status + payloads_by_session.pop(current_session_id, None) proofs: list[dict[str, Any]] = [] sessions: list[dict[str, Any]] = [] diff --git a/backend/api/routes/openrouter.py b/backend/api/routes/openrouter.py index db0830b..0b27ec4 100644 --- a/backend/api/routes/openrouter.py +++ b/backend/api/routes/openrouter.py @@ -423,8 +423,8 @@ async def set_free_model_settings(request: FreeModelSettings) -> Dict[str, Any]: } except RuntimeSettingsError as e: free_model_manager.configure( - looping=bool(previous_status.get("looping_enabled", True)), - auto_selector=bool(previous_status.get("auto_selector_enabled", True)), + looping=bool(previous_status.get("looping_enabled", False)), + auto_selector=bool(previous_status.get("auto_selector_enabled", False)), ) logger.error(f"Failed to persist free model settings: {e}") raise HTTPException(status_code=500, detail="Failed to persist free model settings") diff --git a/backend/api/routes/proof_search.py b/backend/api/routes/proof_search.py index ac9782f..dbe0e49 100644 --- a/backend/api/routes/proof_search.py +++ b/backend/api/routes/proof_search.py @@ -3,13 +3,18 @@ import logging -from fastapi import APIRouter, HTTPException +from fastapi import APIRouter, HTTPException, Query from backend.shared.proof_search.models import ( ProofSearchCorpus, PublicProofSearchRequest, UnifiedProofSearchRecord, ) +from backend.shared.proof_search.assistant_coordinator import assistant_proof_search_coordinator +from backend.shared.proof_search.assistant_models import ( + AssistantSupportLineageResponse, + LatestAssistantProofPackResponse, +) from backend.shared.proof_search.search_service import proof_search_service logger = logging.getLogger(__name__) @@ -28,6 +33,55 @@ async def get_proof_search_overview(): return overview.model_dump(mode="json") +@router.get("/assistant/latest-pack", response_model=LatestAssistantProofPackResponse) +async def get_latest_assistant_proof_pack(): + """Return the latest metadata-only Assistant proof-memory pack for the workflow panel.""" + try: + return assistant_proof_search_coordinator.get_latest_pack_payload() + except Exception as exc: + logger.exception("Failed to load latest Assistant proof pack") + raise HTTPException(status_code=500, detail=f"Assistant proof pack failed: {exc}") from exc + + +@router.get( + "/assistant/targets/{target_hash}/supports/{support_search_id}/lineage", + response_model=AssistantSupportLineageResponse, +) +async def get_assistant_support_lineage( + target_hash: str, + support_search_id: str, + offset: int = Query(default=0, ge=0), + limit: int = Query(default=50, ge=1, le=100), +): + """Return bounded metadata-only provenance for one Assistant support.""" + pack = assistant_proof_search_coordinator.get_latest_pack(target_hash) + support = next( + (item for item in pack.results if item.search_id == support_search_id), + None, + ) if pack else None + if support is None: + raise HTTPException(status_code=404, detail="Assistant target or support not found") + total, occurrences = await proof_search_service.support_lineage( + theorem_statement_hash=support.theorem_statement_hash, + lean_code_hash=support.lean_code_hash, + corpora=[support.corpus], + exclude_run_ids=[support.run_id] if support.run_id else [], + exclude_session_ids=[support.session_id] if support.session_id else [], + offset=offset, + limit=limit, + ) + payload = { + "target_hash": target_hash, + "support_search_id": support_search_id, + "occurrence_total": total, + "offset": offset, + "limit": limit, + "next_offset": offset + len(occurrences) if offset + len(occurrences) < total else None, + "occurrences": occurrences, + } + return payload + + @router.post("/search") async def search_proofs(request: PublicProofSearchRequest): """Search up to seven combined proof records across indexed corpora.""" @@ -44,6 +98,8 @@ async def get_proof_search_record( source: ProofSearchCorpus, proof_id: str, session_id: str | None = None, + search_id: str | None = None, + run_id: str | None = None, ): """Return one indexed proof record, hydrating SyntheticLib4 code when available.""" try: @@ -51,6 +107,8 @@ async def get_proof_search_record( corpus=source, proof_id=proof_id, session_id=session_id, + search_id=search_id, + run_id=run_id, ) except ValueError as exc: logger.warning("Proof-search record hydration rejected: %s", exc) diff --git a/backend/api/routes/proofs.py b/backend/api/routes/proofs.py index efbb5e9..7f28fa2 100644 --- a/backend/api/routes/proofs.py +++ b/backend/api/routes/proofs.py @@ -8,7 +8,7 @@ import time import uuid from pathlib import Path -from typing import Optional, Tuple +from typing import Literal, Optional, Tuple from fastapi import APIRouter, BackgroundTasks, HTTPException, Query from fastapi.responses import JSONResponse, PlainTextResponse @@ -26,6 +26,12 @@ from backend.autonomous.memory.brainstorm_memory import BrainstormMemory, brainstorm_memory from backend.autonomous.memory.paper_library import paper_library from backend.autonomous.memory.proof_database import ProofDatabase, manual_proof_database, proof_database +from backend.autonomous.memory.proof_database import ( + is_duplicate_novel_tier, + is_not_novel_tier, + is_prompt_injection_novel_tier, + normalize_proof_library_category, +) from backend.autonomous.memory.research_metadata import research_metadata from backend.compiler.core.compiler_coordinator import compiler_coordinator from backend.compiler.memory.manual_prompt import load_manual_compiler_prompt @@ -42,6 +48,9 @@ from backend.shared.models import ( ModelConfig, ProofCheckRequest, + ProofCertificateResponse, + ProofLibraryEntry, + ProofLibraryResponse, ProofRoleConfigSnapshot, ProofRuntimeConfigSnapshot, ProofSettingsUpdateRequest, @@ -61,6 +70,7 @@ MANUAL_COMPILER_CURRENT_SOURCE_ID = "manual_compiler_current" PROOF_SCOPE_AUTONOMOUS = "autonomous" PROOF_SCOPE_MANUAL = "manual" +ProofLibraryCategory = Literal["novel", "duplicate_novel", "not_novel", "all"] _manual_proof_run_lock = asyncio.Lock() _LEAN_STATUS_STARTING_LOG_INTERVAL_SECONDS = 60.0 _last_lean_status_starting_log_at = 0.0 @@ -164,9 +174,11 @@ def _attempt_message(prefix: str) -> str: if event_type == "proof_check_started": return "Proof check started for the manual Aggregator database" if event_type == "proof_check_no_candidates": - return "No formal theorem candidates found in the manual Aggregator database" + return "Proof discovery found 0 proof candidates; no proofs will be attempted" if event_type == "proof_check_candidates_found": - return f"Proof candidates found: {data.get('count') or 0}" + count = int(data.get("count") or 0) + subject = "proof candidate" if count == 1 else "proof candidates" + return f"Proof discovery found {count} {subject}; {count} will be attempted" if event_type == "proof_attempt_started": return f"Lean proof attempt started: {target}" if event_type == "proof_lean_accepted": @@ -218,6 +230,39 @@ def _get_scoped_proof_database(scope: str = PROOF_SCOPE_AUTONOMOUS) -> ProofData return proof_database +def _normalize_proof_response_provenance(proof) -> dict: + """Normalize legacy provenance at response time without mutating stored records.""" + if hasattr(proof, "model_dump"): + payload = proof.model_dump(mode="json") + else: + payload = dict(proof or {}) + run_id = str(payload.get("run_id") or payload.get("session_id") or "").strip() + if not run_id: + source_type = str(payload.get("source_type") or "proof").strip() + source_id = str(payload.get("source_id") or payload.get("proof_id") or "legacy").strip() + run_id = f"legacy:{source_type}:{source_id}" + payload["run_id"] = run_id + payload["user_prompt"] = str( + payload.get("user_prompt") + or payload.get("source_title") + or payload.get("theorem_statement") + or "" + ) + tier = str(payload.get("novelty_tier") or "").strip().lower() + if not tier: + tier = "novel_formulation" if bool(payload.get("novel")) else "not_novel" + payload["novelty_tier"] = tier + payload["independent_novelty_tier"] = str( + payload.get("independent_novelty_tier") or tier + ) + payload["independent_novelty_reasoning"] = str( + payload.get("independent_novelty_reasoning") + or payload.get("novelty_reasoning") + or "" + ) + return payload + + def _get_request_proof_database(request: ProofCheckRequest) -> ProofDatabase: if ( (request.source_type == "brainstorm" and request.source_id == MANUAL_AGGREGATOR_SOURCE_ID) @@ -238,17 +283,6 @@ async def _warm_start() -> None: asyncio.create_task(_warm_start()) -def _schedule_lean4_warm_start(client) -> None: - """Warm the Lean workspace without blocking a settings/status request.""" - async def _warm_start() -> None: - try: - await client.warm_start() - except Exception as exc: # pragma: no cover - defensive background task - logger.warning("Lean 4 client warm start failed: %s", exc) - - asyncio.create_task(_warm_start()) - - def _safe_path_label(path_value: str) -> str: """Return a display-safe basename instead of an absolute local path.""" text = str(path_value or "").strip() @@ -280,6 +314,26 @@ async def _get_export_lean_code( raise HTTPException(status_code=404, detail="Proof not found") +async def _get_archived_export( + session_id: str, + proof_id: str, + scope: str, +) -> tuple[dict, str]: + normalized_scope = (scope or PROOF_SCOPE_AUTONOMOUS).strip().lower() + if normalized_scope == PROOF_SCOPE_MANUAL: + payload = await manual_proof_database.get_library_proof_from_history( + _manual_proof_history_root(), session_id, proof_id + ) + elif normalized_scope == PROOF_SCOPE_AUTONOMOUS: + payload = await proof_database.get_library_proof(session_id, proof_id) + else: + raise HTTPException(status_code=400, detail="Proof scope must be 'autonomous' or 'manual'.") + if payload is None: + raise HTTPException(status_code=404, detail="Proof not found") + normalized = _normalize_proof_response_provenance(payload) + return normalized, str(normalized.get("lean_code") or "") + + def _build_model_config(role: ProofRoleConfigSnapshot) -> ModelConfig: return ModelConfig( provider=role.provider, @@ -514,13 +568,13 @@ async def _refresh_manual_assistant_memory( source_content: str, user_prompt: str, ) -> None: - """Run Try-to-Prove Assistant memory even before proof prompt preflight. + """Schedule Try-to-Prove Assistant memory before proof prompt preflight. Manual proof discovery may fail during mandatory-source context validation before it reaches ``api_client_manager.generate_completion()``, so the normal central Assistant injection hook never fires. This preflight refresh - keeps the user-triggered proof-check button covered by Assistant memory and - leaves a visible log/event trail. + keeps the user-triggered proof-check button covered by Assistant memory + without delaying mandatory proof work. """ if not system_config.agent_conversation_memory_enabled: logger.info( @@ -530,6 +584,7 @@ async def _refresh_manual_assistant_memory( ) return + run_id = await manual_proof_database.get_or_create_active_run_id() snapshot = AssistantTargetSnapshot( workflow_mode="manual_proof_check", target_kind="proof_candidate", @@ -545,21 +600,22 @@ async def _refresh_manual_assistant_memory( source_title=source_title, source_type=f"manual_{source_type}", source_id=source_id, + run_id=run_id, source_titles=[source_title] if source_title else [], imports=["Mathlib"], ) logger.info( - "Assistant memory preflight starting for manual proof check %s:%s (%s)", + "Assistant memory preflight scheduling for manual proof check %s:%s (%s)", source_type, source_id, source_title or "untitled source", ) - pack = await assistant_proof_search_coordinator.refresh_now(snapshot) + target_hash = assistant_proof_search_coordinator.submit_target(snapshot) logger.info( - "Assistant memory preflight complete for manual proof check %s:%s (results=%s)", + "Assistant memory preflight scheduled for manual proof check %s:%s (target=%s)", source_type, source_id, - len(pack.results) if pack else 0, + target_hash[:12], ) @@ -818,6 +874,20 @@ async def _run_manual_proof_check(request: ProofCheckRequest) -> None: request, scoped_proof_database, ) + if request.source_id == MANUAL_AGGREGATOR_SOURCE_ID: + canonical_user_prompt = await _manual_aggregator_prompt() + elif request.source_id == MANUAL_COMPILER_CURRENT_SOURCE_ID: + canonical_user_prompt = ( + compiler_coordinator.user_prompt or await load_manual_compiler_prompt() + ) + elif ":" in request.source_id and request.source_type == "paper": + session_id, paper_id = request.source_id.split(":", 1) + history_paper = await paper_library.get_history_paper(session_id, paper_id) + canonical_user_prompt = str( + (history_paper or {}).get("user_prompt", "") or "" + ) + else: + canonical_user_prompt = await research_metadata.get_user_prompt() snapshot = await _get_runtime_snapshot(request) if snapshot is None: if _is_non_appending_manual_source(request): @@ -848,6 +918,8 @@ async def _run_manual_proof_check(request: ProofCheckRequest) -> None: source_type=request.source_type, source_id=request.source_id, user_prompt=user_prompt, + canonical_user_prompt=canonical_user_prompt, + run_id=await scoped_proof_database.get_or_create_active_run_id(), submitter_model=role_config.model_id, submitter_context=role_config.context_window, submitter_max_tokens=role_config.max_output_tokens, @@ -898,7 +970,7 @@ async def list_proofs(scope: str = Query(default=PROOF_SCOPE_AUTONOMOUS)): scoped_proof_database = _get_scoped_proof_database(scope) proofs = await scoped_proof_database.get_all_proofs() return { - "proofs": [proof.model_dump(mode="json") for proof in proofs], + "proofs": [_normalize_proof_response_provenance(proof) for proof in proofs], "counts": scoped_proof_database.count_proofs(), "scope": (scope or PROOF_SCOPE_AUTONOMOUS).strip().lower(), } @@ -910,7 +982,7 @@ async def list_novel_proofs(scope: str = Query(default=PROOF_SCOPE_AUTONOMOUS)): scoped_proof_database = _get_scoped_proof_database(scope) proofs = await scoped_proof_database.get_all_proofs(novel_only=True) return { - "proofs": [proof.model_dump(mode="json") for proof in proofs], + "proofs": [_normalize_proof_response_provenance(proof) for proof in proofs], "counts": scoped_proof_database.count_proofs(), "scope": (scope or PROOF_SCOPE_AUTONOMOUS).strip().lower(), } @@ -922,7 +994,7 @@ async def list_known_proofs(scope: str = Query(default=PROOF_SCOPE_AUTONOMOUS)): scoped_proof_database = _get_scoped_proof_database(scope) proofs = await scoped_proof_database.get_all_proofs(novel_only=False) return { - "proofs": [proof.model_dump(mode="json") for proof in proofs], + "proofs": [_normalize_proof_response_provenance(proof) for proof in proofs], "counts": scoped_proof_database.count_proofs(), "scope": (scope or PROOF_SCOPE_AUTONOMOUS).strip().lower(), } @@ -1249,35 +1321,64 @@ async def run_manual_proof_check(request: ProofCheckRequest, background_tasks: B } -@router.get("/library") +@router.get("/library", response_model=ProofLibraryResponse) async def get_proof_library( - novel_only: bool = True, + novel_only: Optional[bool] = None, + category: Optional[ProofLibraryCategory] = Query(default=None), scope: str = Query(default=PROOF_SCOPE_AUTONOMOUS), ): """Return archived proofs for the selected proof-library scope.""" normalized_scope = (scope or PROOF_SCOPE_AUTONOMOUS).strip().lower() + normalized_category = normalize_proof_library_category(category, novel_only) if normalized_scope == PROOF_SCOPE_MANUAL: + all_proofs = await manual_proof_database.list_proof_library_from_history( + _manual_proof_history_root(), + novel_only=None, + category="all", + ) proofs = await manual_proof_database.list_proof_library_from_history( _manual_proof_history_root(), novel_only=novel_only, + category=normalized_category, ) elif normalized_scope == PROOF_SCOPE_AUTONOMOUS: - proofs = await proof_database.list_proof_library(novel_only=novel_only) + all_proofs = await proof_database.list_proof_library( + novel_only=None, + category="all", + ) + proofs = await proof_database.list_proof_library( + novel_only=novel_only, + category=normalized_category, + ) else: raise HTTPException(status_code=400, detail="Proof scope must be 'autonomous' or 'manual'.") - novel_count = sum(1 for p in proofs if p.get("novel")) + novel_count = sum( + 1 for p in all_proofs + if p.get("novel") and is_prompt_injection_novel_tier(p.get("novelty_tier", "")) + ) + duplicate_novel_count = sum( + 1 for p in all_proofs if is_duplicate_novel_tier(p.get("novelty_tier", "")) + ) + not_novel_count = sum( + 1 for p in all_proofs if is_not_novel_tier(p.get("novelty_tier", "not_novel")) + ) + normalized_all = [_normalize_proof_response_provenance(p) for p in all_proofs] + normalized_proofs = [_normalize_proof_response_provenance(p) for p in proofs] return { - "proofs": proofs, + "proofs": normalized_proofs, "counts": { - "total": len(proofs) if not novel_only else None, - "listed": len(proofs), + "total": len(normalized_all), + "listed": len(normalized_proofs), "novel": novel_count, + "duplicate_novel": duplicate_novel_count, + "not_novel": not_novel_count, }, "scope": normalized_scope, + "category": normalized_category, } -@router.get("/library/{session_id}/{proof_id}") +@router.get("/library/{session_id}/{proof_id}", response_model=ProofLibraryEntry) async def get_library_proof( session_id: str, proof_id: str, @@ -1297,18 +1398,36 @@ async def get_library_proof( raise HTTPException(status_code=400, detail="Proof scope must be 'autonomous' or 'manual'.") if proof is None: raise HTTPException(status_code=404, detail="Proof not found") - return proof + return _normalize_proof_response_provenance(proof) -@router.get("/{proof_id}/certificate") -async def get_proof_certificate( +@router.get("/library/{session_id}/{proof_id}/certificate", response_model=ProofCertificateResponse) +async def get_library_proof_certificate( + session_id: str, proof_id: str, scope: str = Query(default=PROOF_SCOPE_AUTONOMOUS), ): - """Return a machine-readable proof certificate JSON payload.""" - scoped_proof_database = _get_scoped_proof_database(scope) - proof = await _get_export_proof_or_404(proof_id, scoped_proof_database) + """Export an archived proof certificate keyed by its validated run and proof IDs.""" + proof, lean_code = await _get_archived_export(session_id, proof_id, scope) + return await _certificate_response(proof, lean_code) + + +@router.get("/library/{session_id}/{proof_id}/certificate.lean") +async def get_library_proof_certificate_lean( + session_id: str, + proof_id: str, + scope: str = Query(default=PROOF_SCOPE_AUTONOMOUS), +): + """Export archived Lean source keyed by its validated run and proof IDs.""" + proof, lean_code = await _get_archived_export(session_id, proof_id, scope) + return PlainTextResponse( + content=lean_code, + headers={"Content-Disposition": f'attachment; filename="{proof["proof_id"]}.lean"'}, + ) + +async def _certificate_response(proof, lean_code: str) -> JSONResponse: + """Build the common typed certificate payload for live and archived proofs.""" lean_version = "" mathlib_commit = "" if system_config.lean4_enabled: @@ -1318,32 +1437,59 @@ async def get_proof_certificate( mathlib_commit = client.get_mathlib_commit() except (asyncio.TimeoutError, Exception) as exc: logger.warning("Lean 4 certificate metadata lookup timed out or failed: %s", exc) + normalized = _normalize_proof_response_provenance(proof) + payload = ProofCertificateResponse( + proof_id=normalized["proof_id"], + theorem_statement=normalized["theorem_statement"], + theorem_name=normalized.get("theorem_name", ""), + lean_code=lean_code, + solver=normalized.get("solver") or "Lean 4", + lean_version=lean_version, + mathlib_commit=mathlib_commit, + verified_at=( + normalized["created_at"].isoformat() + if normalized.get("created_at") and hasattr(normalized["created_at"], "isoformat") + else str(normalized.get("created_at") or "") or None + ), + source_type=normalized.get("source_type", ""), + source_id=normalized.get("source_id", ""), + source_title=normalized.get("source_title", ""), + run_id=normalized["run_id"], + user_prompt=normalized["user_prompt"], + novel=bool(normalized.get("novel")), + novelty_tier=normalized["novelty_tier"], + novelty_reasoning=normalized.get("novelty_reasoning", ""), + independent_novelty_tier=normalized["independent_novelty_tier"], + independent_novelty_reasoning=normalized["independent_novelty_reasoning"], + exact_duplicate_proof_id=normalized.get("exact_duplicate_proof_id", ""), + exact_duplicate_run_id=normalized.get("exact_duplicate_run_id", ""), + artifact_purpose=normalized.get("artifact_purpose") or "verified_occurrence", + canonical_identity_version=normalized.get("canonical_identity_version", ""), + canonical_theorem_statement_hash=normalized.get( + "canonical_theorem_statement_hash", + "", + ), + canonical_lean_code_hash=normalized.get("canonical_lean_code_hash", ""), + attempt_count=normalized.get("attempt_count") or 0, + solver_hints=list(normalized.get("solver_hints") or []), + dependencies=list(normalized.get("dependencies") or []), + ) + return JSONResponse(content=payload.model_dump(mode="json")) + + +@router.get("/{proof_id}/certificate", response_model=ProofCertificateResponse) +async def get_proof_certificate( + proof_id: str, + scope: str = Query(default=PROOF_SCOPE_AUTONOMOUS), +): + """Return a machine-readable proof certificate JSON payload.""" + scoped_proof_database = _get_scoped_proof_database(scope) + proof = await _get_export_proof_or_404(proof_id, scoped_proof_database) lean_code = await _get_export_lean_code(proof_id, scoped_proof_database) - payload = { - "proof_id": proof.proof_id, - "theorem_statement": proof.theorem_statement, - "theorem_name": proof.theorem_name, - "lean_code": lean_code, - "solver": proof.solver or "Lean 4", - "lean_version": lean_version, - "mathlib_commit": mathlib_commit, - "verified_at": proof.created_at.isoformat() if proof.created_at else None, - "source_type": proof.source_type, - "source_id": proof.source_id, - "source_title": proof.source_title, - "novel": proof.novel, - "novelty_reasoning": proof.novelty_reasoning, - "attempt_count": proof.attempt_count, - "solver_hints": list(proof.solver_hints or []), - "dependencies": [dependency.model_dump(mode="json") for dependency in (proof.dependencies or [])], - } - return JSONResponse( - content=payload, - headers={ - "Content-Disposition": f'attachment; filename="{proof_id}_certificate.json"', - }, - ) + response = await _certificate_response(proof, lean_code) + response.headers["Content-Disposition"] = f'attachment; filename="{proof_id}_certificate.json"' + return response @router.get("/{proof_id}/certificate.lean") @@ -1477,4 +1623,4 @@ async def get_proof( proof = await scoped_proof_database.get_proof(proof_id) if proof is None: raise HTTPException(status_code=404, detail="Proof not found") - return proof.model_dump(mode="json") + return _normalize_proof_response_provenance(proof) diff --git a/backend/api/routes/workflow.py b/backend/api/routes/workflow.py index e84244d..441a6a6 100644 --- a/backend/api/routes/workflow.py +++ b/backend/api/routes/workflow.py @@ -2,13 +2,121 @@ API routes for workflow management. """ from fastapi import APIRouter, HTTPException -from typing import Dict, Any, List +from typing import Dict, Any, List, Literal +from pydantic import BaseModel, Field import logging router = APIRouter() logger = logging.getLogger(__name__) +class SolutionPathStepResponse(BaseModel): + step_id: str + title: str + description: str = "" + status: Literal["pending", "active", "complete", "blocked"] = "pending" + + +class SolutionPathRepairResponse(BaseModel): + proposal_id: str + reason: str = "unknown_reviewer_failure" + detail: str = "" + lifecycle_generation: int + + +class SolutionPathResponse(BaseModel): + success: bool = True + enabled: bool = False + ownership: Literal["none", "active", "resumable"] = "none" + mode: Literal["idle", "aggregator", "compiler", "autonomous", "leanoj"] = "idle" + run_id: str | None = None + lifecycle_generation: int | None = None + acceptance_count: int = 0 + revision: int | None = None + main_route: str = "" + ordering: Literal["ordered", "unordered"] = "ordered" + steps: List[SolutionPathStepResponse] = Field(default_factory=list) + pending_proposals: int = 0 + queued_proposals: int = 0 + reviewing_proposals: int = 0 + repair_required_proposals: int = 0 + repairs: List[SolutionPathRepairResponse] = Field(default_factory=list) + repair_reason: str | None = None + repair_detail: str = "" + message: str = "No solution path is available for the active workflow." + + +class ResumeSolutionPathProposalRequest(BaseModel): + run_id: str = Field(min_length=1, max_length=256) + proposal_id: str = Field(min_length=1, max_length=256) + lifecycle_generation: int = Field(ge=1) + + +class ResumeSolutionPathProposalResponse(BaseModel): + success: bool = True + run_id: str + proposal_id: str + proposal_status: Literal["queued"] + lifecycle_generation: int + message: str + + +def _solution_path_owner(): + """Resolve one explicit active or resumable owner without guessing by load order.""" + from backend.aggregator.core.coordinator import coordinator + from backend.compiler.core.compiler_coordinator import compiler_coordinator + from backend.autonomous.core.autonomous_coordinator import autonomous_coordinator + from backend.leanoj.core.leanoj_coordinator import leanoj_coordinator + from backend.shared.solution_path import solution_path_registry + + coordinators = ( + ("leanoj", leanoj_coordinator, bool(getattr(leanoj_coordinator, "is_active", False))), + ("autonomous", autonomous_coordinator, bool(getattr(autonomous_coordinator, "is_active", False))), + ("compiler", compiler_coordinator, bool(getattr(compiler_coordinator, "is_running", False))), + ("aggregator", coordinator, bool(getattr(coordinator, "is_running", False))), + ) + attached = [] + for mode, owner, active in coordinators: + manager = next( + ( + getattr(owner, name, None) + for name in ("solution_path_manager", "_solution_path_manager") + if getattr(owner, name, None) is not None + ), + None, + ) + if manager is not None: + attached.append((mode, manager, active)) + + active = [(mode, manager) for mode, manager, is_active in attached if is_active] + if len({id(manager) for _, manager in active}) == 1 and active: + return active[0][0], active[0][1], "active" + if active: + return "idle", None, "none" + + loaded = solution_path_registry.loaded_managers() + if len(loaded) == 1: + manager = loaded[0] + workflow_mode = str(getattr(manager.state, "workflow_mode", "") or "") + mode = workflow_mode if workflow_mode in {"autonomous", "leanoj"} else "aggregator" + return mode, manager, "resumable" + if len(loaded) > 1: + manager = solution_path_registry.latest_loaded_manager() + if manager is not None: + workflow_mode = str(getattr(manager.state, "workflow_mode", "") or "") + mode = workflow_mode if workflow_mode in {"autonomous", "leanoj"} else "aggregator" + return mode, manager, "resumable" + return "idle", None, "none" + + unique_attached = {} + for mode, manager, _ in attached: + unique_attached.setdefault(id(manager), (mode, manager)) + if len(unique_attached) == 1: + mode, manager = next(iter(unique_attached.values())) + return mode, manager, "resumable" + return "idle", None, "none" + + def _apply_boost_state(tasks: List[Dict]) -> List[Dict]: """Apply current boost state to tasks before returning to frontend.""" from backend.shared.boost_manager import boost_manager @@ -81,6 +189,134 @@ async def get_workflow_predictions() -> Dict[str, Any]: raise HTTPException(status_code=500, detail="Failed to get predictions") +@router.get("/api/workflow/solution-path", response_model=SolutionPathResponse) +async def get_solution_path() -> SolutionPathResponse: + """Return an in-memory solution-path snapshot without filesystem or model work.""" + try: + mode, manager, ownership = _solution_path_owner() + + state = getattr(manager, "state", None) + plan = getattr(state, "plan", None) + route = getattr(plan, "route", None) + if manager is None or state is None: + return SolutionPathResponse( + mode=mode, + ownership=ownership, + message="Solution path tracking is not available for this workflow yet.", + ) + + statuses = [ + str(getattr(proposal, "status", "")).split(".")[-1].lower() + for proposal in getattr(state, "proposals", []) + ] + queued = sum(status in {"queued", "followup"} for status in statuses) + reviewing = sum(status == "reviewing" for status in statuses) + repair_proposals = [ + proposal + for proposal, status in zip(getattr(state, "proposals", []), statuses) + if status == "user_repair_required" + ] + repairs = [ + SolutionPathRepairResponse( + proposal_id=str(proposal.proposal_id), + reason=str(getattr(proposal, "repair_reason", None) or "unknown_reviewer_failure").split(".")[-1].lower(), + detail=str(getattr(proposal, "repair_detail", "") or getattr(proposal, "feedback", "")), + lifecycle_generation=int( + getattr(proposal, "repair_generation", None) + or getattr(state, "lifecycle_generation", 1) + ), + ) + for proposal in repair_proposals + ] + pending = sum( + 1 + for status in statuses + if status in {"queued", "reviewing", "followup"} + ) + if plan is None or route is None: + return SolutionPathResponse( + enabled=bool(getattr(manager, "active", False)), + ownership=ownership, + mode=mode, + run_id=getattr(state, "run_id", None), + lifecycle_generation=getattr(state, "lifecycle_generation", None), + acceptance_count=getattr(state, "acceptance_count", 0), + pending_proposals=pending, + queued_proposals=queued, + reviewing_proposals=reviewing, + repair_required_proposals=len(repairs), + repairs=repairs, + repair_reason=repairs[0].reason if repairs else None, + repair_detail=repairs[0].detail if repairs else "", + message="Solution path tracking is loaded; no approved plan is available yet. This is normal and some runs may never generate a solution path.", + ) + return SolutionPathResponse( + enabled=True, + ownership=ownership, + mode=mode, + run_id=getattr(state, "run_id", None), + lifecycle_generation=getattr(state, "lifecycle_generation", None), + acceptance_count=getattr(state, "acceptance_count", 0), + revision=getattr(plan, "revision", None), + main_route=str(getattr(plan, "main_route", "")), + ordering=str(getattr(route, "ordering", "ordered")).split(".")[-1].lower(), + steps=[ + SolutionPathStepResponse( + step_id=str(step.step_id), + title=str(step.title), + description=str(getattr(step, "description", "")), + status=str(getattr(step, "status", "pending")), + ) + for step in getattr(route, "steps", []) + ], + pending_proposals=pending, + queued_proposals=queued, + reviewing_proposals=reviewing, + repair_required_proposals=len(repairs), + repairs=repairs, + repair_reason=repairs[0].reason if repairs else None, + repair_detail=repairs[0].detail if repairs else "", + message="Current Main Submitter 1-approved solution path.", + ) + except Exception: + logger.exception("Failed to read solution path snapshot") + return SolutionPathResponse(message="Solution path is temporarily unavailable.") + + +@router.post( + "/api/workflow/solution-path/resume", + response_model=ResumeSolutionPathProposalResponse, +) +async def resume_solution_path_proposal( + request: ResumeSolutionPathProposalRequest, +) -> ResumeSolutionPathProposalResponse: + """Retry one repair-blocked proposal owned by the active/resumable workflow.""" + mode, manager, ownership = _solution_path_owner() + if manager is None or ownership == "none": + raise HTTPException(status_code=404, detail="No active or resumable solution path was found") + state = manager.state + if state.run_id != request.run_id: + raise HTTPException(status_code=409, detail="Solution path run ownership changed") + if state.lifecycle_generation != request.lifecycle_generation: + raise HTTPException(status_code=409, detail="Solution path lifecycle generation changed") + try: + proposal = await manager.resume_proposal( + request.proposal_id, + lifecycle_generation=request.lifecycle_generation, + ) + except ValueError as exc: + detail = str(exc) + status = 404 if "not found" in detail else 409 + raise HTTPException(status_code=status, detail=detail) from exc + return ResumeSolutionPathProposalResponse( + run_id=request.run_id, + proposal_id=proposal.proposal_id, + proposal_status="queued", + lifecycle_generation=request.lifecycle_generation, + message=f"Solution-path update resumed for {mode} settings.", + ) + + @router.get("/api/token-stats") async def get_token_stats() -> Dict[str, Any]: """Return cumulative token usage stats and elapsed research time.""" diff --git a/backend/autonomous/agents/completion_reviewer.py b/backend/autonomous/agents/completion_reviewer.py index 5a92d67..1027c01 100644 --- a/backend/autonomous/agents/completion_reviewer.py +++ b/backend/autonomous/agents/completion_reviewer.py @@ -211,6 +211,13 @@ async def _generate_assessment( # Validate prompt size before sending prompt_tokens = count_tokens(prompt) max_input_tokens = rag_config.get_available_input_tokens(self.context_window, self.max_output_tokens) + from backend.shared.solution_path.integration import with_budgeted_solver_plan + prompt = with_budgeted_solver_plan( + prompt, + getattr(self, "solution_path_manager", None), + max_input_tokens, + ) + prompt_tokens = count_tokens(prompt) task_id = self.get_current_task_id() await api_client_manager.prewarm_assistant_memory_context( @@ -262,6 +269,10 @@ async def _generate_assessment( if decision not in ["continue_brainstorm", "write_paper"]: logger.error(f"CompletionReviewer: Invalid decision: {decision}") return None + reasoning = data.get("reasoning") + if not isinstance(reasoning, str) or not reasoning.strip(): + logger.error("CompletionReviewer: Missing non-empty reasoning") + return None # Notify task completed successfully if self.task_tracking_callback: @@ -269,7 +280,7 @@ async def _generate_assessment( return CompletionReviewResult( decision=decision, - reasoning=data.get("reasoning", "No reasoning provided"), + reasoning=reasoning.strip(), suggested_additions=data.get("suggested_additions", "") ) @@ -319,6 +330,10 @@ async def _self_validate( brainstorm_database=brainstorm_context, # Use prepared context (may be RAG) original_assessment=original_assessment ) + from backend.shared.solution_path.integration import with_validator_hook + prompt = with_validator_hook( + prompt, getattr(self, "solution_path_manager", None) + ) # Validate prompt size before sending prompt_tokens = count_tokens(prompt) @@ -366,16 +381,23 @@ async def _self_validate( # Parse JSON using central utility (handles sanitization + parsing + array handling + enhanced logging) try: data = parse_json(content) - - # Get validation result - check for string "true"/"false" as well - validated_raw = data.get("validated", False) - if isinstance(validated_raw, str): - validated = validated_raw.lower() == "true" - logger.warning(f"CompletionReviewer: 'validated' was a string '{validated_raw}', converted to bool: {validated}") - else: - validated = bool(validated_raw) - - reasoning = data.get("reasoning", "No reasoning") + from backend.shared.solution_path.integration import enqueue_optional_update + validated = data.get("validated") + reasoning = data.get("reasoning") + if type(validated) is not bool: + logger.error("CompletionReviewer: 'validated' must be an exact boolean") + return False + if not isinstance(reasoning, str) or not reasoning.strip(): + logger.error("CompletionReviewer: Self-validation requires non-empty reasoning") + return False + await enqueue_optional_update( + data, + getattr(self, "solution_path_manager", None), + proposer_role=self.role_id, + source_task_id=task_id, + source_phase="completion_self_validation", + source_decision="accept" if validated else "reject", + ) # Notify task completed successfully if self.task_tracking_callback: diff --git a/backend/autonomous/agents/final_answer/answer_format_selector.py b/backend/autonomous/agents/final_answer/answer_format_selector.py index eb8e1bb..5b62aeb 100644 --- a/backend/autonomous/agents/final_answer/answer_format_selector.py +++ b/backend/autonomous/agents/final_answer/answer_format_selector.py @@ -202,6 +202,11 @@ async def _generate_selection( # Validate prompt size prompt_tokens = count_tokens(prompt) max_input = self._calculate_max_input_tokens() + from backend.shared.solution_path.integration import with_budgeted_solver_plan + prompt = with_budgeted_solver_plan( + prompt, getattr(self, "solution_path_manager", None), max_input + ) + prompt_tokens = count_tokens(prompt) if prompt_tokens > max_input: logger.error(f"AnswerFormatSelector: Prompt too large ({prompt_tokens} > {max_input})") @@ -238,15 +243,19 @@ async def _generate_selection( # Parse JSON using central utility data = parse_json(content) - # Validate answer_format value - answer_format = data.get("answer_format", "short_form") - if answer_format not in ["short_form", "long_form"]: - logger.warning(f"AnswerFormatSelector: Invalid format '{answer_format}', defaulting to short_form") - answer_format = "short_form" + answer_format = data.get("answer_format") + reasoning = data.get("reasoning") + if answer_format not in {"short_form", "long_form"}: + raise ValueError( + "Format selection requires answer_format to be exactly " + "'short_form' or 'long_form'" + ) + if not isinstance(reasoning, str) or not reasoning.strip(): + raise ValueError("Format selection requires non-empty reasoning") return AnswerFormatSelection( answer_format=answer_format, - reasoning=data.get("reasoning", "") + reasoning=reasoning.strip(), ) except FreeModelExhaustedError: @@ -278,6 +287,10 @@ async def _validate_selection( certainty_assessment=certainty_assessment.model_dump(), format_selection=selection.model_dump() ) + from backend.shared.solution_path.integration import with_validator_hook + prompt = with_validator_hook( + prompt, getattr(self, "solution_path_manager", None) + ) # Validate prompt size prompt_tokens = count_tokens(prompt) @@ -319,11 +332,23 @@ async def _validate_selection( # Parse JSON using central utility data = parse_json(content) + from backend.shared.solution_path.integration import enqueue_optional_update + decision = data.get("decision") + reasoning = data.get("reasoning") + if decision not in {"accept", "reject"}: + return False, "Validator response requires decision to be exactly accept or reject" + if not isinstance(reasoning, str) or not reasoning.strip(): + return False, "Validator response requires non-empty reasoning" + await enqueue_optional_update( + data, + getattr(self, "solution_path_manager", None), + proposer_role=f"{self.role_id}_validator", + source_task_id=task_id, + source_phase="answer_format_validation", + source_decision=decision, + ) - decision = data.get("decision", "reject") - reasoning = data.get("reasoning", "No reasoning provided") - - return decision == "accept", reasoning + return decision == "accept", reasoning.strip() except FreeModelExhaustedError: raise diff --git a/backend/autonomous/agents/final_answer/certainty_assessor.py b/backend/autonomous/agents/final_answer/certainty_assessor.py index 65ebf49..f3bace6 100644 --- a/backend/autonomous/agents/final_answer/certainty_assessor.py +++ b/backend/autonomous/agents/final_answer/certainty_assessor.py @@ -162,7 +162,8 @@ async def assess_certainty( is_valid, feedback = await self._validate_assessment( user_research_prompt, all_papers, - assessment + assessment, + expanded_papers, ) if is_valid: @@ -272,7 +273,10 @@ def _build_expansion_request_prompt( YOUR TASK: Review the paper abstracts and outlines below. Decide which papers you need to see in FULL CONTENT -to accurately assess what can be answered with certainty (no speculation, no hand-waving). +to accurately assess the strongest defensible answer and the evidence status of its components. +Expand papers when full details are needed to evaluate a mechanism, evidence or derivation, +implementation, risk or limitation, validation method, theorem, or proof. +Do not treat a proposed invention or experiment as demonstrated without the required evidence. You may expand as many papers as needed for a thorough assessment. If the abstracts/outlines provide sufficient information, you may proceed without expansion. @@ -419,6 +423,11 @@ async def _generate_assessment( if prompt_tokens > max_input: logger.error("CertaintyAssessor: Cannot fit even summary-only prompt") return None + from backend.shared.solution_path.integration import with_budgeted_solver_plan + prompt = with_budgeted_solver_plan( + prompt, getattr(self, "solution_path_manager", None), max_input + ) + prompt_tokens = count_tokens(prompt) task_id = self.get_current_task_id() await api_client_manager.prewarm_assistant_memory_context( @@ -457,11 +466,28 @@ async def _generate_assessment( # Parse JSON using central utility data = parse_json(content) - + if not isinstance(data, dict): + raise ValueError("Certainty assessment must be a JSON object") + certainty_level = data.get("certainty_level") + summary = data.get("known_certainties_summary") + reasoning = data.get("reasoning") + if certainty_level not in { + "total_answer", + "partial_answer", + "no_answer_known", + "appears_impossible", + "other", + }: + raise ValueError("Certainty assessment has an invalid or missing certainty_level") + if not isinstance(summary, str) or not summary.strip(): + raise ValueError("Certainty assessment requires a non-empty known_certainties_summary") + if not isinstance(reasoning, str) or not reasoning.strip(): + raise ValueError("Certainty assessment requires non-empty reasoning") + return CertaintyAssessment( - certainty_level=data.get("certainty_level", "other"), - known_certainties_summary=data.get("known_certainties_summary", ""), - reasoning=data.get("reasoning", "") + certainty_level=certainty_level, + known_certainties_summary=summary.strip(), + reasoning=reasoning.strip(), ) except FreeModelExhaustedError: @@ -476,7 +502,8 @@ async def _validate_assessment( self, user_research_prompt: str, all_papers: List[Dict[str, Any]], - assessment: CertaintyAssessment + assessment: CertaintyAssessment, + expanded_papers: List[Dict[str, Any]] = None, ) -> tuple[bool, str]: """ Validate the certainty assessment. @@ -489,7 +516,12 @@ async def _validate_assessment( prompt = build_certainty_validation_prompt( user_research_prompt=user_research_prompt, papers_summary=all_papers, - assessment=assessment.model_dump() + assessment=assessment.model_dump(), + expanded_papers=expanded_papers, + ) + from backend.shared.solution_path.integration import with_validator_hook + prompt = with_validator_hook( + prompt, getattr(self, "solution_path_manager", None) ) # Validate prompt size @@ -532,8 +564,16 @@ async def _validate_assessment( # Parse JSON using central utility data = parse_json(content) - + from backend.shared.solution_path.integration import enqueue_optional_update decision = data.get("decision", "reject") + await enqueue_optional_update( + data, + getattr(self, "solution_path_manager", None), + proposer_role=f"{self.role_id}_validator", + source_task_id=task_id, + source_phase="certainty_validation", + source_decision=decision if decision in {"accept", "reject"} else None, + ) reasoning = data.get("reasoning", "No reasoning provided") return decision == "accept", reasoning diff --git a/backend/autonomous/agents/final_answer/volume_organizer.py b/backend/autonomous/agents/final_answer/volume_organizer.py index b65f7f8..6011672 100644 --- a/backend/autonomous/agents/final_answer/volume_organizer.py +++ b/backend/autonomous/agents/final_answer/volume_organizer.py @@ -184,19 +184,10 @@ async def organize_volume( current_volume = organization.model_dump() validator_feedback = feedback - # Force completion on max iterations - logger.warning(f"VolumeOrganizer: Forcing completion at iteration {self.MAX_ITERATIONS}") - - if current_volume: - organization = VolumeOrganization( - volume_title=current_volume.get("volume_title", "Research Volume"), - chapters=[VolumeChapter(**ch) for ch in current_volume.get("chapters", [])], - outline_complete=True, - revision_reasoning="Forced completion after maximum iterations" - ) - await final_answer_memory.save_volume_organization(organization) - return organization - + logger.error( + "VolumeOrganizer: No validator-approved completed organization after " + f"{self.MAX_ITERATIONS} iterations" + ) return None async def _generate_organization( @@ -230,6 +221,11 @@ async def _generate_organization( # Validate prompt size prompt_tokens = count_tokens(prompt) max_input = self._calculate_max_input_tokens() + from backend.shared.solution_path.integration import with_budgeted_solver_plan + prompt = with_budgeted_solver_plan( + prompt, getattr(self, "solution_path_manager", None), max_input + ) + prompt_tokens = count_tokens(prompt) if prompt_tokens > max_input: logger.error(f"VolumeOrganizer: Prompt too large ({prompt_tokens} > {max_input})") @@ -265,48 +261,80 @@ async def _generate_organization( # Parse JSON using central utility data = parse_json(content) - - # Parse chapters + volume_title = data.get("volume_title") + outline_complete = data.get("outline_complete") + reasoning = data.get("reasoning") + chapter_items = data.get("chapters") + if not isinstance(volume_title, str) or not volume_title.strip(): + raise ValueError("Volume organization requires a non-empty volume_title") + if type(outline_complete) is not bool: + raise ValueError("Volume organization requires outline_complete to be a boolean") + if not isinstance(reasoning, str) or not reasoning.strip(): + raise ValueError("Volume organization requires non-empty reasoning") + if not isinstance(chapter_items, list) or not chapter_items: + raise ValueError("Volume organization requires a non-empty chapters list") + chapters = [] - for ch in data.get("chapters", []): + for index, ch in enumerate(chapter_items, start=1): + if not isinstance(ch, dict): + raise ValueError(f"Volume chapter {index} must be an object") + chapter_type = ch.get("chapter_type") + title = ch.get("title") + order = ch.get("order") + status = ch.get("status", "pending") + description = ch.get("description", "") + paper_id = ch.get("paper_id") + if chapter_type not in { + "existing_paper", "introduction", "conclusion", "gap_paper" + }: + raise ValueError(f"Volume chapter {index} has invalid chapter_type") + if not isinstance(title, str) or not title.strip(): + raise ValueError(f"Volume chapter {index} requires a non-empty title") + if type(order) is not int or order < 1: + raise ValueError(f"Volume chapter {index} requires a positive integer order") + if status not in {"pending", "writing", "complete"}: + raise ValueError(f"Volume chapter {index} has invalid status") + if not isinstance(description, str): + raise ValueError(f"Volume chapter {index} description must be a string") + if chapter_type == "existing_paper" and ( + not isinstance(paper_id, str) or not paper_id.strip() + ): + raise ValueError( + f"Existing-paper chapter {index} requires a non-empty paper_id" + ) chapter = VolumeChapter( - chapter_type=ch.get("chapter_type", "existing_paper"), - paper_id=ch.get("paper_id"), - title=ch.get("title", "Untitled Chapter"), - order=ch.get("order", len(chapters) + 1), - status=ch.get("status", "pending"), - description=ch.get("description", "") + chapter_type=chapter_type, + paper_id=paper_id.strip() if isinstance(paper_id, str) else None, + title=title.strip(), + order=order, + status=status, + description=description.strip(), ) chapters.append(chapter) - # Ensure we have introduction and conclusion - has_intro = any(ch.chapter_type == "introduction" for ch in chapters) - has_conclusion = any(ch.chapter_type == "conclusion" for ch in chapters) - - if not has_intro: - chapters.insert(0, VolumeChapter( - chapter_type="introduction", - title="Introduction", - order=1, - description="Introduction to the volume" - )) - - if not has_conclusion: - chapters.append(VolumeChapter( - chapter_type="conclusion", - title="Conclusion", - order=len(chapters) + 1, - description="Conclusion of the volume" - )) - - # Fix ordering if needed - chapters = self._normalize_chapter_order(chapters) + intro_count = sum(ch.chapter_type == "introduction" for ch in chapters) + conclusion_count = sum(ch.chapter_type == "conclusion" for ch in chapters) + if intro_count != 1 or conclusion_count != 1: + raise ValueError( + "Volume organization requires exactly one introduction and one conclusion" + ) + orders = [ch.order for ch in chapters] + if sorted(orders) != list(range(1, len(chapters) + 1)): + raise ValueError("Volume chapter orders must be unique and contiguous from 1") + ordered = sorted(chapters, key=lambda chapter: chapter.order) + if ( + ordered[0].chapter_type != "introduction" + or ordered[-1].chapter_type != "conclusion" + ): + raise ValueError( + "Volume organization must place introduction first and conclusion last" + ) return VolumeOrganization( - volume_title=data.get("volume_title", "Research Volume"), - chapters=chapters, - outline_complete=data.get("outline_complete", False), - revision_reasoning=data.get("reasoning", "") + volume_title=volume_title.strip(), + chapters=ordered, + outline_complete=outline_complete, + revision_reasoning=reasoning.strip(), ) except FreeModelExhaustedError: @@ -371,6 +399,10 @@ async def _validate_organization( papers_summary=all_papers, volume_organization=organization.model_dump() ) + from backend.shared.solution_path.integration import with_validator_hook + prompt = with_validator_hook( + prompt, getattr(self, "solution_path_manager", None) + ) # Validate prompt size prompt_tokens = count_tokens(prompt) @@ -412,8 +444,16 @@ async def _validate_organization( # Parse JSON using central utility data = parse_json(content) - + from backend.shared.solution_path.integration import enqueue_optional_update decision = data.get("decision", "reject") + await enqueue_optional_update( + data, + getattr(self, "solution_path_manager", None), + proposer_role=f"{self.role_id}_validator", + source_task_id=task_id, + source_phase="volume_organization_validation", + source_decision=decision if decision in {"accept", "reject"} else None, + ) reasoning = data.get("reasoning", "No reasoning provided") return decision == "accept", reasoning diff --git a/backend/autonomous/agents/paper_title_selector.py b/backend/autonomous/agents/paper_title_selector.py index e1794e4..aaac084 100644 --- a/backend/autonomous/agents/paper_title_selector.py +++ b/backend/autonomous/agents/paper_title_selector.py @@ -265,6 +265,12 @@ async def _generate_title( raise ValueError( "Title generation prompt exceeds context limit even after shedding optional title context." ) + from backend.shared.solution_path.integration import with_budgeted_solver_plan + prompt = with_budgeted_solver_plan( + prompt, + getattr(self, "solution_path_manager", None), + max_input_tokens, + ) self.task_sequence += 1 @@ -296,18 +302,22 @@ async def _generate_title( # Parse JSON using central utility data = parse_json(content) - title = data.get("paper_title", "") - if not title: + title = data.get("paper_title") + reasoning = data.get("reasoning") + if not isinstance(title, str) or not title.strip(): logger.error("PaperTitleSelector: No title in response") return None + if not isinstance(reasoning, str) or not reasoning.strip(): + logger.error("PaperTitleSelector: Missing non-empty reasoning") + return None # Notify task completed successfully if self.task_tracking_callback: self.task_tracking_callback("completed", task_id) return PaperTitleSelection( - paper_title=title, - reasoning=data.get("reasoning", "") + paper_title=title.strip(), + reasoning=reasoning.strip(), ) except FreeModelExhaustedError: @@ -347,6 +357,10 @@ async def _validate_title( proposed_title=proposed_title, title_reasoning=title_reasoning ) + from backend.shared.solution_path.integration import with_validator_hook + prompt = with_validator_hook( + prompt, getattr(self, "solution_path_manager", None) + ) max_input_tokens = rag_config.get_available_input_tokens( self.validator_context_window, @@ -390,9 +404,21 @@ async def _validate_title( # Parse JSON using central utility data = parse_json(content) - + from backend.shared.solution_path.integration import enqueue_optional_update decision = data.get("decision", "").lower() - reasoning = data.get("reasoning", "No reasoning provided") + reasoning = data.get("reasoning") + if decision not in {"accept", "reject"}: + return False, "Title validator requires decision to be exactly accept or reject" + if not isinstance(reasoning, str) or not reasoning.strip(): + return False, "Title validator requires non-empty reasoning" + await enqueue_optional_update( + data, + getattr(self, "solution_path_manager", None), + proposer_role="paper_title_validator", + source_task_id=task_id, + source_phase="paper_title_validation", + source_decision=decision, + ) # Notify task completed successfully if self.task_tracking_callback: @@ -401,7 +427,7 @@ async def _validate_title( if decision == "accept": return True, "" else: - return False, reasoning + return False, reasoning.strip() except FreeModelExhaustedError: raise diff --git a/backend/autonomous/agents/proof_formalization_agent.py b/backend/autonomous/agents/proof_formalization_agent.py index 06a2858..4f7a3ce 100644 --- a/backend/autonomous/agents/proof_formalization_agent.py +++ b/backend/autonomous/agents/proof_formalization_agent.py @@ -15,14 +15,15 @@ from backend.shared.model_error_utils import ( format_transient_provider_error, is_non_retryable_model_error, + is_provider_context_length_error, is_retryable_model_output_error, is_transient_model_call_error, ) from backend.shared.models import ProofAttemptFeedback, ProofCandidate, SmtHint from backend.shared.openrouter_client import FreeModelExhaustedError +from backend.shared.provider_errors import ProviderContextLengthError from backend.shared.proof_search.tool_adapter import execute_search_lean_proofs from backend.shared.proof_search.assistant_coordinator import assistant_proof_search_coordinator -from backend.shared.proof_search.assistant_models import AssistantTargetSnapshot from backend.shared.utils import count_tokens from backend.shared.config import rag_config, system_config from backend.autonomous.prompts.proof_prompts import ( @@ -33,16 +34,6 @@ logger = logging.getLogger(__name__) -def _assistant_workflow_mode_for_role(role_id: str) -> str: - normalized = (role_id or "").lower() - if "manual" in normalized or "compiler_aggregator" in normalized: - return "manual_proof_check" - if normalized.startswith("compiler") or normalized.startswith("comp_"): - return "compiler" - if normalized.startswith("leanoj"): - return "leanoj" - return "autonomous" - AttemptCallback = Callable[[ProofAttemptFeedback], Awaitable[None]] AttemptStartCallback = Callable[[int, str], Awaitable[None]] ShouldStopFn = Optional[Callable[[], bool]] @@ -68,15 +59,92 @@ def _assistant_workflow_mode_for_role(role_id: str) -> str: "upstream provider timeout", ) _MALFORMED_MODEL_OUTPUT_REASON = "Model returned malformed output (not valid JSON); retrying with clean context." -_INCOMPLETE_MODEL_OUTPUT_ERROR = ( - "MODEL OUTPUT INCOMPLETE: provider stopped before returning usable proof output " - "(max_output_tokens). Preserve the proof checkpoint and retry with adjusted output budget or prompt size." +_INCOMPLETE_MODEL_OUTPUT_REASON = ( + "Model/provider output reached its maximum output length and was truncated before returning usable Lean proof code." ) _LEAN_WORKSPACE_ERROR_PREFIX = "LEAN 4 WORKSPACE ERROR" _MANDATORY_FULL_SOURCE_CONTEXT_OVERFLOW_PREFIX = "MANDATORY FULL SOURCE CONTEXT OVERFLOW" _PROOF_SEARCH_CONTEXT_OMITTED = ( "[Proof-search context omitted because it was unavailable or did not fit the configured context budget.]" ) +_TRUNCATED_FINISH_REASONS = { + "incomplete", + "length", + "max_tokens", + "max_output_tokens", + "token_limit", + "output_limit", +} +_TRUNCATED_RESPONSE_STATUSES = { + "incomplete", + "length", + "max_tokens", + "max_output_tokens", +} + + +class ProofFormalizationContextOverflowError(ValueError): + """Compiler-facing overflow that retains the failed proof route.""" + + def __init__(self, feedback: ProofAttemptFeedback) -> None: + super().__init__(feedback.error_output) + self.feedback = feedback + + +def _truncated_model_output_feedback( + theorem_candidate: ProofCandidate, + *, + attempt_number: int, + strategy: str, +) -> ProofAttemptFeedback: + return ProofAttemptFeedback( + attempt=attempt_number, + theorem_id=theorem_candidate.theorem_id, + reasoning=_INCOMPLETE_MODEL_OUTPUT_REASON, + lean_code="", + error_output=( + "MODEL OUTPUT TRUNCATED: the selected model/provider reached its maximum output length " + "before returning usable Lean proof code. This attempt is counted as failed; " + "research will continue with the next proof attempt or candidate." + ), + goal_states="", + strategy=strategy, + success=False, + ) + + +def _latest_assistant_pack_for_lean_attempts() -> tuple[str, str, list[dict[str, Any]]]: + """Reuse the latest Assistant proof context without refreshing during Lean attempts.""" + assistant_pack = assistant_proof_search_coordinator.get_latest_reusable_pack() + if not assistant_pack or not assistant_pack.results: + return "", "", [] + return ( + assistant_pack.target_hash, + assistant_pack.to_prompt_context(), + [support.model_dump(mode="json") for support in assistant_pack.results], + ) + + +def _response_indicates_output_truncation(response: dict[str, Any]) -> bool: + if not isinstance(response, dict): + return False + choices = response.get("choices") + if isinstance(choices, list) and choices: + first_choice = choices[0] if isinstance(choices[0], dict) else {} + finish_reason = str(first_choice.get("finish_reason") or "").strip().lower() + if finish_reason in _TRUNCATED_FINISH_REASONS: + return True + status = str(response.get("status") or "").strip().lower() + if status in _TRUNCATED_RESPONSE_STATUSES: + return True + incomplete_details = response.get("incomplete_details") + if isinstance(incomplete_details, dict): + reason = str(incomplete_details.get("reason") or "").strip().lower() + if reason in _TRUNCATED_FINISH_REASONS: + return True + return False + + def _is_stop_requested(should_stop: ShouldStopFn) -> bool: if should_stop is None: return False @@ -138,6 +206,45 @@ def _is_context_overflow_feedback(feedback: ProofAttemptFeedback) -> bool: ) +def _is_provider_context_overflow(exc: Exception) -> bool: + return isinstance(exc, ProviderContextLengthError) or is_provider_context_length_error(exc) + + +def _provider_context_overflow_feedback( + theorem_candidate: ProofCandidate, + *, + attempt_number: int, + strategy: str, + context_window: int, + max_output_tokens: int, + exc: Exception, +) -> ProofAttemptFeedback: + typed_error = isinstance(exc, ProviderContextLengthError) + route = getattr(exc, "route", None) if typed_error else None + safe_detail = str(getattr(exc, "safe_message", "") or "").strip() if typed_error else str(exc) + return ProofAttemptFeedback( + attempt=attempt_number, + theorem_id=theorem_candidate.theorem_id, + reasoning="Provider rejected the proof prompt because it exceeded the model context window.", + lean_code="", + error_output=( + f"{_MANDATORY_FULL_SOURCE_CONTEXT_OVERFLOW_PREFIX}: Provider rejected the proof " + "formalization prompt before generation because the input exceeded the selected " + f"model context window. Configured total context={context_window}, " + f"max output reserve={max_output_tokens}. Lean 4 was not run." + + (f" Provider detail: {safe_detail}" if safe_detail else "") + ), + goal_states="", + strategy=strategy, + success=False, + configured_model=getattr(route, "configured_model", None), + configured_provider=getattr(route, "configured_provider", None), + effective_model=getattr(route, "model", None), + effective_provider=getattr(route, "provider", None), + overflow_origin="provider", + ) + + class ProofFormalizationAgent: """Turn theorem candidates into Lean 4 code and retry with feedback.""" @@ -357,6 +464,9 @@ async def _run_full_script_attempt( ), strategy="full_script", success=False, + overflow_origin="local_preflight", + prompt_tokens=prompt_tokens, + max_input_tokens=max_input_tokens, ) return "", source_excerpt, feedback @@ -378,6 +488,21 @@ async def _run_full_script_attempt( role_id=self.role_id, task_id=task_id, ) + if _response_indicates_output_truncation(response): + logger.warning( + "ProofFormalizationAgent full-script attempt %s for %s returned a length-truncated provider response.", + attempt_number, + theorem_candidate.theorem_id, + ) + return ( + "", + source_excerpt, + _truncated_model_output_feedback( + theorem_candidate, + attempt_number=attempt_number, + strategy="full_script", + ), + ) if not response or not response.get("choices"): raise ValueError("Empty response from formalization model.") @@ -418,10 +543,40 @@ async def _run_full_script_attempt( except RetryableProviderError: raise except Exception as exc: + if _is_provider_context_overflow(exc): + feedback = _provider_context_overflow_feedback( + theorem_candidate, + attempt_number=attempt_number, + strategy="full_script", + context_window=self.context_window, + max_output_tokens=self.max_output_tokens, + exc=exc, + ) + logger.warning( + "ProofFormalizationAgent full-script attempt %s context overflow for %s: %s", + attempt_number, + theorem_candidate.theorem_id, + exc, + ) + return "", source_excerpt, feedback if is_non_retryable_model_error(exc): raise if is_retryable_model_output_error(exc): - raise RuntimeError(_INCOMPLETE_MODEL_OUTPUT_ERROR) from exc + logger.warning( + "ProofFormalizationAgent full-script attempt %s for %s hit provider output truncation: %s", + attempt_number, + theorem_candidate.theorem_id, + exc, + ) + return ( + "", + source_excerpt, + _truncated_model_output_feedback( + theorem_candidate, + attempt_number=attempt_number, + strategy="full_script", + ), + ) if is_transient_model_call_error(exc): raise RetryableProviderError( provider="unknown", @@ -476,29 +631,8 @@ async def prove_candidate( theorem_candidate.statement, source_content, ) - assistant_snapshot = AssistantTargetSnapshot( - workflow_mode=_assistant_workflow_mode_for_role(self.role_id), - target_kind="proof_candidate", - user_prompt=user_research_prompt, - target_statement=theorem_candidate.statement, - formal_sketch=theorem_candidate.formal_sketch, - proof_attempt_feedback=_format_attempt_feedback_for_assistant(attempts), - source_title=source_title, - source_type=source_type, - source_id=theorem_candidate.origin_source_id, - dependency_names=[ - str(getattr(lemma, "full_name", "") or getattr(lemma, "requested_name", "") or "").strip() - for lemma in (theorem_candidate.relevant_lemmas or []) - if str(getattr(lemma, "full_name", "") or getattr(lemma, "requested_name", "") or "").strip() - ], - ) - assistant_target_hash = assistant_proof_search_coordinator.submit_target(assistant_snapshot) - assistant_pack = assistant_proof_search_coordinator.get_latest_pack(assistant_target_hash) - retrieved_proofs_context = assistant_pack.to_prompt_context() if assistant_pack else "" - retrieved_proof_records: list[dict[str, Any]] = ( - [support.model_dump(mode="json") for support in assistant_pack.results] - if assistant_pack - else [] + assistant_target_hash, retrieved_proofs_context, retrieved_proof_records = ( + _latest_assistant_pack_for_lean_attempts() ) theorem_name = "" @@ -520,17 +654,6 @@ async def prove_candidate( theorem_candidate.theorem_id, ) break - if attempts: - assistant_snapshot = assistant_snapshot.model_copy( - update={"proof_attempt_feedback": _format_attempt_feedback_for_assistant(attempts)} - ) - assistant_target_hash = assistant_proof_search_coordinator.submit_target(assistant_snapshot) - assistant_pack = assistant_proof_search_coordinator.get_latest_pack(assistant_target_hash) - if assistant_pack: - retrieved_proofs_context = assistant_pack.to_prompt_context() - retrieved_proof_records = [ - support.model_dump(mode="json") for support in assistant_pack.results - ] attempt_number = next_attempt_number + attempt_offset if attempt_start_callback and malformed_output_retries == 0: await attempt_start_callback(attempt_number, "full_script") @@ -546,7 +669,7 @@ async def prove_candidate( smt_hint=smt_hint, source_title=source_title, retrieved_proofs_context=retrieved_proofs_context, - assistant_memory_target_hash=assistant_target_hash if assistant_pack and assistant_pack.results else "", + assistant_memory_target_hash=assistant_target_hash, ) terminal_malformed_output = False @@ -611,29 +734,8 @@ async def prove_candidate_tactic_script( theorem_candidate.statement, source_content, ) - assistant_snapshot = AssistantTargetSnapshot( - workflow_mode=_assistant_workflow_mode_for_role(self.role_id), - target_kind="proof_candidate", - user_prompt=user_research_prompt, - target_statement=theorem_candidate.statement, - formal_sketch=theorem_candidate.formal_sketch, - proof_attempt_feedback=_format_attempt_feedback_for_assistant(attempts), - source_title=source_title, - source_type=source_type, - source_id=theorem_candidate.origin_source_id, - dependency_names=[ - str(getattr(lemma, "full_name", "") or getattr(lemma, "requested_name", "") or "").strip() - for lemma in (theorem_candidate.relevant_lemmas or []) - if str(getattr(lemma, "full_name", "") or getattr(lemma, "requested_name", "") or "").strip() - ], - ) - assistant_target_hash = assistant_proof_search_coordinator.submit_target(assistant_snapshot) - assistant_pack = assistant_proof_search_coordinator.get_latest_pack(assistant_target_hash) - retrieved_proofs_context = assistant_pack.to_prompt_context() if assistant_pack else "" - retrieved_proof_records: list[dict[str, Any]] = ( - [support.model_dump(mode="json") for support in assistant_pack.results] - if assistant_pack - else [] + assistant_target_hash, retrieved_proofs_context, retrieved_proof_records = ( + _latest_assistant_pack_for_lean_attempts() ) theorem_name = "" @@ -658,18 +760,6 @@ async def prove_candidate_tactic_script( attempt_number = next_attempt_number + attempt_offset if attempt_start_callback and malformed_output_retries == 0: await attempt_start_callback(attempt_number, "tactic_script") - if attempts: - assistant_snapshot = assistant_snapshot.model_copy( - update={"proof_attempt_feedback": _format_attempt_feedback_for_assistant(attempts)} - ) - assistant_target_hash = assistant_proof_search_coordinator.submit_target(assistant_snapshot) - assistant_pack = assistant_proof_search_coordinator.get_latest_pack(assistant_target_hash) - if assistant_pack: - retrieved_proofs_context = assistant_pack.to_prompt_context() - retrieved_proof_records = [ - support.model_dump(mode="json") for support in assistant_pack.results - ] - prompt, source_excerpt, max_input_tokens, prompt_tokens = self._fit_prompt_to_context( build_proof_tactic_script_prompt, min_excerpt_length=1500, @@ -704,6 +794,9 @@ async def prove_candidate_tactic_script( ), strategy="tactic_script", success=False, + overflow_origin="local_preflight", + prompt_tokens=prompt_tokens, + max_input_tokens=max_input_tokens, ) attempts.append(feedback) if attempt_callback: @@ -722,12 +815,28 @@ async def prove_candidate_tactic_script( max_tokens=self.max_output_tokens, temperature=0.0, ) - if assistant_target_hash and assistant_pack and assistant_pack.results: + if assistant_target_hash: assistant_proof_search_coordinator.mark_pack_consumed_by_solver( assistant_target_hash, role_id=self.role_id, task_id=task_id, ) + if _response_indicates_output_truncation(response): + logger.warning( + "ProofFormalizationAgent tactic-script attempt %s for %s returned a length-truncated provider response.", + attempt_number, + theorem_candidate.theorem_id, + ) + feedback = _truncated_model_output_feedback( + theorem_candidate, + attempt_number=attempt_number, + strategy="tactic_script", + ) + attempts.append(feedback) + if attempt_callback: + await attempt_callback(feedback) + attempt_offset += 1 + continue if not response or not response.get("choices"): raise ValueError("Empty response from tactic formalization model.") @@ -766,7 +875,7 @@ async def prove_candidate_tactic_script( smt_hint=smt_hint, source_title=source_title, retrieved_proofs_context=retrieved_proofs_context, - assistant_memory_target_hash=assistant_target_hash if assistant_pack and assistant_pack.results else "", + assistant_memory_target_hash=assistant_target_hash, ) if current_theorem_name: theorem_name = current_theorem_name @@ -843,10 +952,44 @@ async def prove_candidate_tactic_script( except RetryableProviderError: raise except Exception as exc: + if _is_provider_context_overflow(exc): + feedback = _provider_context_overflow_feedback( + theorem_candidate, + attempt_number=attempt_number, + strategy="tactic_script", + context_window=self.context_window, + max_output_tokens=self.max_output_tokens, + exc=exc, + ) + logger.warning( + "ProofFormalizationAgent tactic-script attempt %s context overflow for %s: %s", + attempt_number, + theorem_candidate.theorem_id, + exc, + ) + attempts.append(feedback) + if attempt_callback: + await attempt_callback(feedback) + break if is_non_retryable_model_error(exc): raise if is_retryable_model_output_error(exc): - raise RuntimeError(_INCOMPLETE_MODEL_OUTPUT_ERROR) from exc + logger.warning( + "ProofFormalizationAgent tactic-script attempt %s for %s hit provider output truncation: %s", + attempt_number, + theorem_candidate.theorem_id, + exc, + ) + feedback = _truncated_model_output_feedback( + theorem_candidate, + attempt_number=attempt_number, + strategy="tactic_script", + ) + attempts.append(feedback) + if attempt_callback: + await attempt_callback(feedback) + attempt_offset += 1 + continue if is_transient_model_call_error(exc): raise RetryableProviderError( provider="unknown", diff --git a/backend/autonomous/agents/proof_identification_agent.py b/backend/autonomous/agents/proof_identification_agent.py index c98c4b7..c23e641 100644 --- a/backend/autonomous/agents/proof_identification_agent.py +++ b/backend/autonomous/agents/proof_identification_agent.py @@ -2,12 +2,13 @@ Proof identification agent for Lean 4 verification checkpoints. """ import logging -from typing import List, Tuple +from typing import Any, Dict, List, Tuple from backend.shared.api_client_manager import RetryableProviderError, api_client_manager -from backend.shared.json_parser import parse_json +from backend.shared.json_parser import parse_json, sanitize_model_output_for_retry_context from backend.shared.response_extraction import extract_message_text from backend.shared.model_error_utils import ( + is_retryable_model_output_error, is_non_retryable_model_error, is_transient_model_call_error, ) @@ -39,16 +40,193 @@ def __init__( context_window: int, max_output_tokens: int, role_id: str, + solution_path_manager: Any = None, ) -> None: self.model_id = model_id self.context_window = context_window self.max_output_tokens = max_output_tokens self.role_id = role_id + self.solution_path_manager = solution_path_manager self.task_sequence = 0 def get_current_task_id(self) -> str: return f"proof_id_{self.task_sequence:03d}" + @staticmethod + def _extract_response_content(response: Dict[str, Any]) -> str: + if not response or not response.get("choices"): + raise ValueError("Proof identification returned no model choices.") + message = response["choices"][0].get("message", {}) + content = extract_message_text(message) + if not content: + raise ValueError("Proof identification returned empty model output.") + return content + + @staticmethod + def _parse_candidate_payload(content: str) -> Tuple[bool, List[ProofCandidate]]: + data = parse_json(content) + if isinstance(data, list): + if not data: + raise ValueError("Proof identification returned an empty JSON array.") + data = data[0] + if not isinstance(data, dict): + raise ValueError("Proof identification returned JSON that was not an object.") + if "has_provable_theorems" not in data: + raise ValueError("Proof identification JSON omitted has_provable_theorems.") + if not isinstance(data.get("has_provable_theorems"), bool): + raise ValueError("Proof identification has_provable_theorems must be a boolean.") + + has_candidates = data["has_provable_theorems"] + raw_theorems_value = data.get("theorems", []) + if raw_theorems_value is None: + raw_theorems_value = [] + if not isinstance(raw_theorems_value, list): + raise ValueError("Proof identification theorems must be an array.") + if has_candidates and not raw_theorems_value: + raise ValueError( + "Proof identification claimed provable theorems but returned no theorem entries." + ) + + theorem_candidates: List[ProofCandidate] = [] + malformed_candidate_count = 0 + non_novel_candidate_count = 0 + for index, theorem in enumerate(raw_theorems_value, start=1): + if not isinstance(theorem, dict): + malformed_candidate_count += 1 + continue + statement = str(theorem.get("statement", "")).strip() + if not statement: + malformed_candidate_count += 1 + continue + theorem_id = theorem.get("theorem_id") or theorem.get("id") or f"thm_{index}" + expected_novelty_tier = str(theorem.get("expected_novelty_tier", "")).strip().lower() + if expected_novelty_tier == "not_novel": + non_novel_candidate_count += 1 + logger.info( + "ProofIdentificationAgent skipped theorem %s because it was marked not_novel.", + theorem_id, + ) + continue + if expected_novelty_tier not in _NOVEL_PROOF_TIERS: + malformed_candidate_count += 1 + logger.info( + "ProofIdentificationAgent skipped theorem %s because it did not include a valid expected_novelty_tier.", + theorem_id, + ) + continue + prompt_relevance_rationale = str( + theorem.get("prompt_relevance_rationale", "") + ).strip() + novelty_rationale = str(theorem.get("novelty_rationale", "")).strip() + why_not_standard_known_result = str( + theorem.get("why_not_standard_known_result", "") + ).strip() + if not ( + prompt_relevance_rationale + and novelty_rationale + and why_not_standard_known_result + ): + malformed_candidate_count += 1 + logger.info( + "ProofIdentificationAgent skipped theorem %s because it lacked required prompt-relevance, novelty, or anti-standard-result rationale.", + theorem_id, + ) + continue + theorem_candidates.append( + ProofCandidate( + theorem_id=str(theorem_id), + statement=statement, + formal_sketch=str(theorem.get("formal_sketch", "")).strip(), + expected_novelty_tier=expected_novelty_tier, + prompt_relevance_rationale=prompt_relevance_rationale, + novelty_rationale=novelty_rationale, + why_not_standard_known_result=why_not_standard_known_result, + ) + ) + + if has_candidates and not theorem_candidates and malformed_candidate_count: + raise ValueError( + "Proof identification claimed provable theorems but returned no valid theorem candidates " + f"({malformed_candidate_count} malformed, {non_novel_candidate_count} not_novel)." + ) + + return has_candidates and bool(theorem_candidates), theorem_candidates + + async def _retry_identification_output( + self, + *, + prompt: str, + task_id: str, + failed_output: str, + error: Exception, + max_input_tokens: int, + ) -> Tuple[bool, List[ProofCandidate]]: + logger.info("ProofIdentificationAgent: initial output failed; attempting bounded JSON retry.") + retry_prompt = ( + "Your previous proof-identification response could not be used.\n\n" + f"ERROR: {error}\n\n" + "Return the same proof-identification decision in valid JSON only. " + "Use this exact top-level shape:\n" + "{\n" + ' "has_provable_theorems": true or false,\n' + ' "theorems": [\n' + " {\n" + ' "theorem_id": "short_id",\n' + ' "statement": "the theorem statement",\n' + ' "formal_sketch": "Lean-relevant proof sketch",\n' + ' "expected_novelty_tier": "major_mathematical_discovery | mathematical_discovery | novel_variant | novel_formulation | not_novel",\n' + ' "prompt_relevance_rationale": "why this directly answers or substantially advances the user prompt",\n' + ' "novelty_rationale": "why this is not merely standard or routine",\n' + ' "why_not_standard_known_result": "why this is not a standard Mathlib/textbook result"\n' + " }\n" + " ]\n" + "}\n\n" + "Respond with ONLY the JSON object, no markdown and no explanation." + ) + prompt_with_retry_instruction = f"{prompt}\n\n---\n{retry_prompt}" + instruction_tokens = count_tokens(prompt_with_retry_instruction) + if instruction_tokens <= max_input_tokens: + messages = [{"role": "user", "content": prompt_with_retry_instruction}] + else: + logger.warning( + "ProofIdentificationAgent retry instruction too large (%s > %s); retrying original prompt.", + instruction_tokens, + max_input_tokens, + ) + messages = [{"role": "user", "content": prompt}] + failed_output_preview = sanitize_model_output_for_retry_context( + failed_output, + max_chars=2000, + ) + if failed_output_preview: + conversation_tokens = ( + count_tokens(prompt) + + count_tokens(failed_output_preview) + + count_tokens(retry_prompt) + ) + if conversation_tokens <= max_input_tokens: + messages = [ + {"role": "user", "content": prompt}, + {"role": "assistant", "content": failed_output_preview}, + {"role": "user", "content": retry_prompt}, + ] + else: + logger.warning( + "ProofIdentificationAgent retry conversation too large (%s > %s); retrying original prompt.", + conversation_tokens, + max_input_tokens, + ) + response = await api_client_manager.generate_completion( + task_id=f"{task_id}_retry", + role_id=self.role_id, + model=self.model_id, + messages=messages, + max_tokens=self.max_output_tokens, + temperature=0.0, + ) + retry_content = self._extract_response_content(response) + return self._parse_candidate_payload(retry_content) + async def translate_candidate_to_smt( self, *, @@ -160,8 +338,12 @@ async def identify_candidates( proof_max_rounds=proof_max_rounds, prior_round_results=prior_round_results, ) - prompt_tokens = count_tokens(prompt) max_input_tokens = rag_config.get_available_input_tokens(self.context_window, self.max_output_tokens) + from backend.shared.solution_path.integration import with_budgeted_solver_plan + prompt = with_budgeted_solver_plan( + prompt, self.solution_path_manager, max_input_tokens + ) + prompt_tokens = count_tokens(prompt) task_id = self.get_current_task_id() await api_client_manager.prewarm_assistant_memory_context( task_id=task_id, @@ -192,104 +374,28 @@ async def identify_candidates( max_tokens=self.max_output_tokens, temperature=0.0, ) - if not response or not response.get("choices"): - raise ValueError("Proof identification returned no model choices.") - - message = response["choices"][0].get("message", {}) - content = extract_message_text(message) - if not content: - raise ValueError("Proof identification returned empty model output.") - - data = parse_json(content) - if isinstance(data, list): - if not data: - raise ValueError("Proof identification returned an empty JSON array.") - data = data[0] - if not isinstance(data, dict): - raise ValueError("Proof identification returned JSON that was not an object.") - if "has_provable_theorems" not in data: - raise ValueError("Proof identification JSON omitted has_provable_theorems.") - if not isinstance(data.get("has_provable_theorems"), bool): - raise ValueError("Proof identification has_provable_theorems must be a boolean.") - - has_candidates = data["has_provable_theorems"] - raw_theorems_value = data.get("theorems", []) - if raw_theorems_value is None: - raw_theorems_value = [] - if not isinstance(raw_theorems_value, list): - raise ValueError("Proof identification theorems must be an array.") - if has_candidates and not raw_theorems_value: - raise ValueError( - "Proof identification claimed provable theorems but returned no theorem entries." + content = self._extract_response_content(response) + try: + return self._parse_candidate_payload(content) + except Exception as parse_error: + return await self._retry_identification_output( + prompt=prompt, + task_id=task_id, + failed_output=content, + error=parse_error, + max_input_tokens=max_input_tokens, ) - raw_theorems = raw_theorems_value - theorem_candidates: List[ProofCandidate] = [] - malformed_candidate_count = 0 - non_novel_candidate_count = 0 - for index, theorem in enumerate(raw_theorems, start=1): - if not isinstance(theorem, dict): - malformed_candidate_count += 1 - continue - statement = str(theorem.get("statement", "")).strip() - if not statement: - malformed_candidate_count += 1 - continue - theorem_id = theorem.get("theorem_id") or theorem.get("id") or f"thm_{index}" - expected_novelty_tier = str(theorem.get("expected_novelty_tier", "")).strip().lower() - if expected_novelty_tier == "not_novel": - non_novel_candidate_count += 1 - logger.info( - "ProofIdentificationAgent skipped theorem %s because it was marked not_novel.", - theorem_id, - ) - continue - if expected_novelty_tier not in _NOVEL_PROOF_TIERS: - malformed_candidate_count += 1 - logger.info( - "ProofIdentificationAgent skipped theorem %s because it did not include a valid expected_novelty_tier.", - theorem_id, - ) - continue - prompt_relevance_rationale = str( - theorem.get("prompt_relevance_rationale", "") - ).strip() - novelty_rationale = str(theorem.get("novelty_rationale", "")).strip() - why_not_standard_known_result = str( - theorem.get("why_not_standard_known_result", "") - ).strip() - if not ( - prompt_relevance_rationale - and novelty_rationale - and why_not_standard_known_result - ): - malformed_candidate_count += 1 - logger.info( - "ProofIdentificationAgent skipped theorem %s because it lacked required prompt-relevance, novelty, or anti-standard-result rationale.", - theorem_id, - ) - continue - theorem_candidates.append( - ProofCandidate( - theorem_id=str(theorem_id), - statement=statement, - formal_sketch=str(theorem.get("formal_sketch", "")).strip(), - expected_novelty_tier=expected_novelty_tier, - prompt_relevance_rationale=prompt_relevance_rationale, - novelty_rationale=novelty_rationale, - why_not_standard_known_result=why_not_standard_known_result, - ) - ) - - if has_candidates and not theorem_candidates and malformed_candidate_count: - raise ValueError( - "Proof identification claimed provable theorems but returned no valid theorem candidates " - f"({malformed_candidate_count} malformed, {non_novel_candidate_count} not_novel)." - ) - - return has_candidates and bool(theorem_candidates), theorem_candidates except FreeModelExhaustedError: raise except Exception as exc: + if is_retryable_model_output_error(exc): + return await self._retry_identification_output( + prompt=prompt, + task_id=task_id, + failed_output="", + error=exc, + max_input_tokens=max_input_tokens, + ) if ( isinstance(exc, RetryableProviderError) or is_non_retryable_model_error(exc) diff --git a/backend/autonomous/agents/reference_selector.py b/backend/autonomous/agents/reference_selector.py index 96af3ca..7f23c60 100644 --- a/backend/autonomous/agents/reference_selector.py +++ b/backend/autonomous/agents/reference_selector.py @@ -73,11 +73,13 @@ def __init__( self, model_id: str, context_window: int = 0, - max_output_tokens: int = 0 + max_output_tokens: int = 0, + solution_path_manager: Optional[Any] = None, ): self.model_id = model_id self.context_window = context_window self.max_output_tokens = max_output_tokens + self.solution_path_manager = solution_path_manager # Task tracking for workflow panel and boost integration self.task_sequence: int = 0 @@ -244,6 +246,10 @@ async def _request_expansion( already_selected_papers=already_selected_papers, max_total_papers=max_total_papers, ) + from backend.shared.solution_path.integration import with_budgeted_solver_plan + prompt = with_budgeted_solver_plan( + prompt, self.solution_path_manager, self._calculate_max_input_tokens() + ) task_id = self.get_current_task_id() await api_client_manager.prewarm_assistant_memory_context( @@ -291,15 +297,33 @@ async def _request_expansion( # Parse JSON using central utility data = parse_json(content) + expand_papers = data.get("expand_papers") + proceed_without_references = data.get("proceed_without_references") + reasoning = data.get("reasoning") + if not isinstance(expand_papers, list) or not all( + isinstance(paper_id, str) and paper_id.strip() + for paper_id in expand_papers + ): + raise ValueError("Reference expansion requires expand_papers to be a string list") + if type(proceed_without_references) is not bool: + raise ValueError( + "Reference expansion requires proceed_without_references to be a boolean" + ) + if not isinstance(reasoning, str) or not reasoning.strip(): + raise ValueError("Reference expansion requires non-empty reasoning") + if proceed_without_references and expand_papers: + raise ValueError( + "Reference expansion cannot request papers while proceeding without references" + ) # Notify task completed successfully if self.task_tracking_callback: self.task_tracking_callback("completed", task_id) return ReferenceExpansionRequest( - expand_papers=data.get("expand_papers", []), - proceed_without_references=data.get("proceed_without_references", False), - reasoning=data.get("reasoning", "") + expand_papers=[paper_id.strip() for paper_id in expand_papers], + proceed_without_references=proceed_without_references, + reasoning=reasoning.strip(), ) except FreeModelExhaustedError: @@ -425,6 +449,10 @@ async def _make_final_selection( max_papers=max_papers, retrieved_context=retrieved_context, ) + from backend.shared.solution_path.integration import with_budgeted_solver_plan + prompt = with_budgeted_solver_plan( + prompt, self.solution_path_manager, max_input + ) task_id = self.get_current_task_id() await api_client_manager.prewarm_assistant_memory_context( @@ -455,6 +483,9 @@ async def _make_final_selection( mode=mode, max_papers=max_papers, ) + prompt = with_budgeted_solver_plan( + prompt, self.solution_path_manager, max_input + ) prompt_tokens = count_tokens(prompt) if prompt_tokens > max_input: logger.error( @@ -501,7 +532,18 @@ async def _make_final_selection( # Parse JSON using central utility data = parse_json(content) - selected = data.get("selected_papers", []) + selected = data.get("selected_papers") + reasoning = data.get("reasoning") + if not isinstance(selected, list) or not all( + isinstance(paper_id, str) and paper_id.strip() + for paper_id in selected + ): + raise ValueError( + "Final reference selection requires selected_papers to be a string list" + ) + if not isinstance(reasoning, str) or not reasoning.strip(): + raise ValueError("Final reference selection requires non-empty reasoning") + selected = [paper_id.strip() for paper_id in selected] # Enforce max papers limit if len(selected) > max_papers: diff --git a/backend/autonomous/agents/topic_selector.py b/backend/autonomous/agents/topic_selector.py index 18d5252..e38673b 100644 --- a/backend/autonomous/agents/topic_selector.py +++ b/backend/autonomous/agents/topic_selector.py @@ -33,7 +33,7 @@ class TopicSelectorAgent: """ Agent that selects the next brainstorm topic. - Can choose to start a new topic, continue existing, or combine topics. + Can choose to start a new topic or continue an existing incomplete topic. Context handling: - Direct injects all metadata summaries (brainstorms, papers, rejections) @@ -130,6 +130,13 @@ async def select_topic( if prompt_tokens > max_input_tokens: logger.error(f"TopicSelector: Even after truncation, prompt ({prompt_tokens}) exceeds limit ({max_input_tokens})") return None + from backend.shared.solution_path.integration import with_budgeted_solver_plan + prompt = with_budgeted_solver_plan( + prompt, + getattr(self, "solution_path_manager", None), + max_input_tokens, + ) + prompt_tokens = count_tokens(prompt) task_id = self.get_current_task_id() await api_client_manager.prewarm_assistant_memory_context( @@ -179,17 +186,20 @@ async def select_topic( # Validate required fields action = data.get("action", "") - if action not in ["new_topic", "continue_existing", "combine_topics"]: + if action not in ["new_topic", "continue_existing"]: logger.error(f"TopicSelector: Invalid action: {action}") return None + reasoning = data.get("reasoning") + if not isinstance(reasoning, str) or not reasoning.strip(): + logger.error("TopicSelector: Missing non-empty reasoning") + return None # Create submission submission = TopicSelectionSubmission( action=action, topic_id=data.get("topic_id"), - topic_ids=data.get("topic_ids", []), topic_prompt=data.get("topic_prompt", ""), - reasoning=data.get("reasoning", "No reasoning provided") + reasoning=reasoning.strip(), ) # Validate based on action @@ -197,11 +207,7 @@ async def select_topic( logger.error("TopicSelector: continue_existing requires topic_id") return None - if action == "combine_topics" and len(submission.topic_ids) < 2: - logger.error("TopicSelector: combine_topics requires at least 2 topic_ids") - return None - - if action in ["new_topic", "combine_topics"] and not submission.topic_prompt: + if action == "new_topic" and not submission.topic_prompt: logger.error(f"TopicSelector: {action} requires topic_prompt") return None @@ -244,7 +250,7 @@ async def handle_rejection( """ await autonomous_rejection_logs.add_topic_selection_rejection( action=submission.action, - proposed_topic=submission.topic_prompt or submission.topic_id or str(submission.topic_ids), + proposed_topic=submission.topic_prompt or submission.topic_id or "", rejection_reasoning=rejection_reasoning ) logger.info(f"TopicSelector: Logged rejection for action={submission.action}") diff --git a/backend/autonomous/agents/topic_validator.py b/backend/autonomous/agents/topic_validator.py index cca367e..e095b61 100644 --- a/backend/autonomous/agents/topic_validator.py +++ b/backend/autonomous/agents/topic_validator.py @@ -93,7 +93,6 @@ async def validate( proposed_action = { "action": submission.action, "topic_id": submission.topic_id, - "topic_ids": submission.topic_ids, "topic_prompt": submission.topic_prompt, "reasoning": submission.reasoning } @@ -105,6 +104,15 @@ async def validate( papers_summary=papers_summary, proposed_action=proposed_action ) + from backend.shared.solution_path.integration import with_validator_hook + prompt = with_validator_hook( + prompt, getattr(self, "solution_path_manager", None) + ) + if override_prompt: + from backend.shared.solution_path.integration import with_validator_hook + prompt = with_validator_hook( + prompt, getattr(self, "solution_path_manager", None) + ) # Validate prompt size prompt_tokens = count_tokens(prompt) @@ -131,6 +139,10 @@ async def validate( papers_summary=truncated_papers, proposed_action=proposed_action ) + from backend.shared.solution_path.integration import with_validator_hook + prompt = with_validator_hook( + prompt, getattr(self, "solution_path_manager", None) + ) prompt_tokens = count_tokens(prompt) if prompt_tokens > max_input_tokens: @@ -172,7 +184,7 @@ async def validate( # Parse JSON using central utility (handles sanitization + parsing + array handling) try: data = parse_json(content) - + from backend.shared.solution_path.integration import enqueue_optional_update # parse_json already handles array response, but keep check for safety if isinstance(data, list) and len(data) > 0: logger.warning("TopicValidator: Model returned array, using first element") @@ -180,15 +192,25 @@ async def validate( # Validate required fields decision = data.get("decision", "").lower() - reasoning = data.get("reasoning", "No reasoning provided") - + reasoning = data.get("reasoning") + if not isinstance(reasoning, str) or not reasoning.strip(): + logger.error("TopicValidator: Missing non-empty reasoning") + return self._create_rejection("Missing non-empty reasoning") if decision not in ["accept", "reject"]: logger.error(f"TopicValidator: Invalid decision: {decision}") return self._create_rejection(f"Invalid decision format: {decision}") + await enqueue_optional_update( + data, + getattr(self, "solution_path_manager", None), + proposer_role="topic_validator", + source_task_id=task_id, + source_phase="topic_validation", + source_decision=decision, + ) result = TopicValidationResult( decision=decision, - reasoning=reasoning + reasoning=reasoning.strip(), ) # Notify task completed successfully diff --git a/backend/autonomous/core/autonomous_coordinator.py b/backend/autonomous/core/autonomous_coordinator.py index 140bdd9..4a59c62 100644 --- a/backend/autonomous/core/autonomous_coordinator.py +++ b/backend/autonomous/core/autonomous_coordinator.py @@ -31,6 +31,7 @@ from backend.shared.json_parser import parse_json from backend.shared.response_extraction import extract_message_text from backend.shared.log_redaction import redact_log_text +from backend.shared.path_safety import resolve_path_within_root, validate_single_path_component from backend.shared.context_overflow import ( CONTEXT_OVERFLOW_STOP_MESSAGE, CONTEXT_OVERFLOW_STOP_REASON, @@ -130,9 +131,13 @@ def __init__(self): self._state = AutonomousResearchState() self._stop_event = asyncio.Event() self._main_task: Optional[asyncio.Task] = None + self.top_level_terminal_callback: Optional[Callable[[], Any]] = None self._stop_broadcast_sent = False self._fatal_stop_reason: Optional[str] = None self._fatal_stop_message: str = "" + self._fatal_stop_payload: Dict[str, Any] = {} + self.solution_path_manager = None + self._ownership_handoff_active = False # Configuration (set during initialize) self._user_research_prompt: str = "" @@ -208,6 +213,9 @@ def __init__(self): self._current_reference_papers: List[str] = [] # Reference papers for current topic cycle self._current_reference_brainstorms: List[str] = [] # Reference brainstorms for proof-only cycles self._acceptance_count: int = 0 + # Session-wide real Tier 1 acceptances; unlike _acceptance_count this + # never resets when topic ownership advances. + self._solution_path_acceptance_count: int = 0 self._rejection_count: int = 0 self._cleanup_removals: int = 0 # Track actual cleanup/pruning removals from aggregator self._consecutive_rejections: int = 0 @@ -259,10 +267,11 @@ async def _broadcast(self, event: str, data: Dict[str, Any] = None) -> None: # broadcast_event expects (event_type, data) as separate arguments await self._broadcast_callback(event, data or {}) - def _mark_context_overflow_stop(self) -> None: + def _mark_context_overflow_stop(self, payload: Optional[Dict[str, Any]] = None) -> None: """Remember that the next stopped event should explain the fatal overflow.""" self._fatal_stop_reason = CONTEXT_OVERFLOW_STOP_REASON self._fatal_stop_message = CONTEXT_OVERFLOW_STOP_MESSAGE + self._fatal_stop_payload = dict(payload or {}) def _track_child_aggregator(self, aggregator: AggregatorCoordinator) -> None: """Track local child aggregators so parent phase changes can stop them.""" @@ -294,12 +303,18 @@ async def _await_parent_phase_shutdown( await asyncio.gather(task, return_exceptions=True) return False - async def _stop_active_child_aggregators(self, reason: str) -> None: + async def _stop_active_child_aggregators( + self, + reason: str, + *, + timeout: float = _PARENT_PHASE_SHUTDOWN_TIMEOUT_SECONDS, + ) -> None: for aggregator in list(self._active_child_aggregators): try: if await self._await_parent_phase_shutdown( f"child aggregator shutdown for {reason}", aggregator.stop(), + timeout=timeout, ): logger.info("Stopped child aggregator for %s", reason) except Exception as exc: @@ -310,7 +325,11 @@ async def _stop_active_child_aggregators(self, reason: str) -> None: def _append_proof_framing(self, prompt: str) -> str: """Append the persisted proof-framing context when active.""" effective_prompt = prompt or "" - if self._proof_framing_active and self._proof_framing_context: + if ( + getattr(self, "_allow_mathematical_proofs", True) + and self._proof_framing_active + and self._proof_framing_context + ): if self._proof_framing_context not in effective_prompt: effective_prompt = f"{effective_prompt}\n\n{self._proof_framing_context}".strip() return effective_prompt @@ -318,6 +337,8 @@ def _append_proof_framing(self, prompt: str) -> str: def _apply_proof_context(self, prompt: str) -> str: """Append proof framing context and inject verified novel proofs.""" effective_prompt = self._append_proof_framing(prompt) + if not self._allow_mathematical_proofs: + return effective_prompt return proof_database.inject_into_prompt(effective_prompt) def _get_effective_user_research_prompt(self) -> str: @@ -327,6 +348,8 @@ def _get_effective_user_research_prompt(self) -> str: async def _get_effective_brainstorm_prompt(self, topic_prompt: str) -> str: """Return the brainstorm prompt with proof context applied.""" effective_prompt = self._apply_proof_context(topic_prompt) + if not self._allow_mathematical_proofs: + return effective_prompt effective_prompt = await proof_database.inject_failure_hints_into_prompt( effective_prompt, self._current_topic_id or "", @@ -347,7 +370,42 @@ async def _get_effective_brainstorm_prompt(self, topic_prompt: str) -> str: def _get_effective_compiler_prompt(self, paper_title: str) -> str: """Return the compiler prompt with proof context applied.""" return self._apply_proof_context( - f"Write a mathematical research paper titled: {paper_title}" + f"Write a rigorous solution paper titled: {paper_title}\n\n" + "Aggressively pursue the strongest credible and genuinely novel solution " + "to the user's exact objective. Use the solution form and verification " + "standard appropriate to the problem's domain and claim types. " + "Mathematical reasoning, theorem " + "discovery, and formal proof remain first-class whenever relevant." + ) + + def _build_tier3_compiler_prompt( + self, + paper_title: str, + findings_summary: str, + writing_context: str = "", + ) -> str: + """Build the proof-aware final-answer paper or chapter prompt.""" + chapter_context = ( + f"\nTier 3 Chapter Role and Required Scope: {writing_context}\n" + if writing_context + else "" + ) + role_instruction = ( + "IMPORTANT: This paper directly advances the final answer by fulfilling " + "the exact volume-chapter role below.\n" + if writing_context + else "IMPORTANT: This paper directly answers the research question.\n" + ) + return self._apply_proof_context( + f"Write a rigorous final-answer solution paper titled: {paper_title}\n\n" + f"{role_instruction}" + "Aggressively pursue the strongest credible and genuinely novel " + "solution using domain- and claim-appropriate rigor. Mathematical " + "reasoning, theorems, and formal proof remain first-class whenever " + "relevant. Preserve the evidence status of every claim; do not " + "present proposals or hypotheses as demonstrated results.\n" + f"Defensible Findings and Evidence Status: {findings_summary}" + f"{chapter_context}" ) def _proof_outputs_enabled(self) -> bool: @@ -464,11 +522,7 @@ async def _run_proof_framing_gate(self) -> None: self._proof_framing_active = is_proof_amenable self._proof_framing_context = PROOF_FRAMING_CONTEXT if is_proof_amenable else "" self._proof_framing_reasoning = reasoning - self._user_research_prompt = ( - self._append_proof_framing(base_prompt) - if is_proof_amenable - else base_prompt - ) + self._user_research_prompt = base_prompt await research_metadata.set_proof_framing_state( base_user_prompt=base_prompt, @@ -665,6 +719,8 @@ async def run_single_proof_round( source_type=source_type, source_id=source_id, user_prompt=self._get_effective_user_research_prompt(), + canonical_user_prompt=self._base_user_research_prompt, + run_id=getattr(session_manager, "session_id", "") or "", submitter_model=submitter_model, submitter_context=submitter_context, submitter_max_tokens=submitter_max_tokens, @@ -688,7 +744,12 @@ async def run_single_proof_round( proof_max_rounds=proof_max_rounds, prior_round_results=prior_round_results, ) - if not self._stop_event.is_set() and not getattr(proof_result, "had_error", False): + has_deferred = bool(getattr(proof_result, "deferred_candidate_ids", [])) + if ( + not self._stop_event.is_set() + and not getattr(proof_result, "had_error", False) + and not has_deferred + ): await research_metadata.mark_proof_checkpoint_trigger_complete( source_type, source_id, @@ -704,6 +765,15 @@ async def run_single_proof_round( ) if getattr(proof_result, "had_error", False): return "error_preserved", proof_result + if has_deferred: + logger.info( + "Proof verification deferred %s context-overflow candidate(s) for %s %s trigger=%s", + len(proof_result.deferred_candidate_ids), + source_type, + source_id, + round_trigger, + ) + return "deferred", proof_result return "completed", proof_result except ProofVerificationProviderPause as exc: retry_candidates = exc.remaining_candidates or retry_candidates @@ -872,6 +942,11 @@ async def run_single_proof_round( ) if round_status in {"stopped", "no_candidates_skipped"}: return "complete" if round_status == "no_candidates_skipped" else "stopped" + if round_status == "deferred": + # Candidate-local overflow is nonfatal to the parent workflow. + # Keep the checkpoint so a later Start with changed proof + # settings retries only deferred/unprocessed candidates. + return "complete" if proof_result is None: continue if self._stop_event.is_set() or getattr(proof_result, "had_error", False): @@ -921,7 +996,12 @@ async def _run_brainstorm_completion_proofs(self) -> str: if self._stop_event.is_set(): return "stopped" - await research_metadata.clear_proof_checkpoint("brainstorm", self._current_topic_id) + proof_checkpoint = await research_metadata.get_proof_checkpoint( + "brainstorm", + self._current_topic_id, + ) + if not proof_checkpoint or proof_checkpoint.get("status") != "deferred": + await research_metadata.clear_proof_checkpoint("brainstorm", self._current_topic_id) await self._save_workflow_state( tier="tier2_paper_writing", phase="pre_paper_compilation", @@ -1157,6 +1237,12 @@ async def initialize( assistant_supercharge_enabled if assistant_model else validator_supercharge_enabled ) self._allow_mathematical_proofs = bool(allow_mathematical_proofs) + if not self._allow_mathematical_proofs: + # The coordinator is a singleton, so a new papers-only run must not + # inherit proof emphasis from a prior proof-enabled run. + self._proof_framing_active = False + self._proof_framing_context = "" + self._proof_framing_reasoning = "Mathematical proof outputs are disabled for this run." self._allow_research_papers = bool(allow_research_papers) self._tier3_enabled = bool(tier3_enabled and self._allow_research_papers) self._creativity_emphasis_boost_enabled = creativity_emphasis_boost_enabled @@ -1244,16 +1330,17 @@ async def initialize( if not self._base_user_research_prompt: self._base_user_research_prompt = self._user_research_prompt - # CRITICAL: Reset and clear all RAG state for fresh autonomous session - # This prevents cross-contamination from Part 1 manual mode - # Autonomous mode should start with a clean RAG that only contains: - # - Brainstorm database (for the current topic) - # - Reference papers (selected by the reference selector) - # Old user uploads or Part 1 aggregator content must be removed - logger.info("Resetting RAG state for fresh autonomous research mode...") - autonomous_rag_manager.reset() # Reset tracking state (indexed sets) - await asyncio.to_thread(rag_manager.clear_all_documents) # Clear all RAG content (non-blocking) - logger.info("RAG state reset and cleared for autonomous mode") + # Chroma is derived state. Windows startup replaces prior-process native + # state before any workflow opens it; session resume then lazily indexes + # the durable sources it needs. A new session also requests an empty + # in-process cache so no previous active workflow sources remain. + autonomous_rag_manager.reset() + if interrupted_session: + logger.info("Autonomous session resume will rebuild required RAG sources lazily") + else: + logger.info("Preparing fresh atomic RAG cache for new autonomous session...") + await rag_manager.clear_all_documents_async() + logger.info("Fresh autonomous RAG cache prepared") # Now initialize with fresh state await autonomous_rag_manager.initialize() @@ -1325,6 +1412,14 @@ async def initialize( validator_context_window=validator_context_window, validator_max_output_tokens=validator_max_tokens, ) + await self._initialize_solution_path_manager() + for agent in ( + self._topic_selector, self._topic_validator, self._completion_reviewer, + self._reference_selector, self._title_selector, self._redundancy_checker, + self._certainty_assessor, self._format_selector, self._volume_organizer, + ): + setattr(agent, "solution_path_manager", self.solution_path_manager) + self._proof_verification_stage.solution_path_manager = self.solution_path_manager # Initialize Tier 3 memory await final_answer_memory.initialize() @@ -1789,6 +1884,10 @@ async def _check_resume_state(self) -> None: self._current_reference_brainstorms = workflow_state.get("reference_brainstorm_ids", []) self._current_paper_title = workflow_state.get("current_paper_title") self._acceptance_count = workflow_state.get("acceptance_count", 0) + self._solution_path_acceptance_count = workflow_state.get( + "solution_path_acceptance_count", + self._acceptance_count, + ) if self._current_topic_id: await self._recover_brainstorm_acceptance_count(self._current_topic_id) self._rejection_count = workflow_state.get("rejection_count", 0) @@ -1803,17 +1902,33 @@ async def _check_resume_state(self) -> None: self._brainstorm_paper_count = workflow_state.get("brainstorm_paper_count", 0) self._current_brainstorm_paper_ids = workflow_state.get("current_brainstorm_paper_ids", []) - # Restore proof framing state - self._proof_framing_active = workflow_state.get("proof_framing_active", False) - self._proof_framing_context = workflow_state.get("proof_framing_context", "") - self._proof_framing_reasoning = workflow_state.get("proof_framing_reasoning", "") + # Allowed Outputs is authoritative on resume. A papers-only restart + # must not revive proof emphasis persisted by an earlier run config. + if self._allow_mathematical_proofs: + self._proof_framing_active = workflow_state.get("proof_framing_active", False) + self._proof_framing_context = workflow_state.get("proof_framing_context", "") + self._proof_framing_reasoning = workflow_state.get("proof_framing_reasoning", "") + else: + self._proof_framing_active = False + self._proof_framing_context = "" + self._proof_framing_reasoning = ( + "Mathematical proof outputs are disabled for this run." + ) self._base_user_research_prompt = await research_metadata.get_base_user_prompt() if not self._base_user_research_prompt: self._base_user_research_prompt = ( workflow_state.get("base_user_research_prompt") or self._user_research_prompt ) - self._user_research_prompt = self._append_proof_framing(self._base_user_research_prompt) + self._user_research_prompt = self._base_user_research_prompt + if not self._allow_mathematical_proofs: + await research_metadata.set_proof_framing_state( + base_user_prompt=self._base_user_research_prompt, + effective_user_prompt=self._user_research_prompt, + active=False, + context="", + reasoning=self._proof_framing_reasoning, + ) # Restore Tier 3 flags for proper resume self._tier3_active = workflow_state.get("tier3_active", False) @@ -2371,6 +2486,7 @@ async def _save_workflow_state(self, tier: str = None, phase: Any = _WORKFLOW_PH "reference_paper_ids": self._current_reference_papers, # Persist reference papers across restarts "reference_brainstorm_ids": self._current_reference_brainstorms, "acceptance_count": self._acceptance_count, + "solution_path_acceptance_count": self._solution_path_acceptance_count, "rejection_count": self._rejection_count, "consecutive_rejections": self._consecutive_rejections, "exhaustion_signals": self._exhaustion_signals, @@ -2384,6 +2500,8 @@ async def _save_workflow_state(self, tier: str = None, phase: Any = _WORKFLOW_PH "proof_framing_active": self._proof_framing_active, "proof_framing_context": self._proof_framing_context, "proof_framing_reasoning": self._proof_framing_reasoning, + "allow_mathematical_proofs": self._allow_mathematical_proofs, + "allow_research_papers": self._allow_research_papers, # Tier 3 Final Answer crash recovery fields "tier3_active": self._tier3_active, "tier3_enabled": self._tier3_enabled, @@ -2452,10 +2570,99 @@ async def _broadcast_stopped_once(self) -> None: "final_stats": stats } if self._fatal_stop_reason: + payload.update(self._fatal_stop_payload) payload["reason"] = self._fatal_stop_reason payload["message"] = self._fatal_stop_message or CONTEXT_OVERFLOW_STOP_MESSAGE await self._broadcast("auto_research_stopped", payload) + async def _initialize_solution_path_manager(self) -> None: + """Create one persistent manager shared by every autonomous phase.""" + from backend.shared.solution_path import ( + build_review_prompt, + compact_review_prompt, + review_with_json_retry, + solution_path_registry, + ) + + primary = next( + ( + config + for config in self._submitter_configs + if config.submitter_id == 1 + ), + None, + ) + if primary is None: + raise ValueError( + "Autonomous Main Submitter 1 is required for solution-path review." + ) + reviewer_role_id = "autonomous_solution_path_reviewer" + api_client_manager.configure_role( + reviewer_role_id, + ModelConfig( + provider=primary.provider, + model_id=primary.model_id, + openrouter_model_id=( + primary.model_id if primary.provider == "openrouter" else None + ), + openrouter_provider=primary.openrouter_provider, + openrouter_reasoning_effort=primary.openrouter_reasoning_effort, + lm_studio_fallback_id=primary.lm_studio_fallback_id, + context_window=primary.context_window, + max_output_tokens=primary.max_output_tokens, + supercharge_enabled=primary.supercharge_enabled, + ), + ) + + async def reviewer(proposal, current_plan): + prompt = build_review_prompt( + user_prompt=self._base_user_research_prompt, + proposal=proposal, + current_plan=current_plan, + ) + task_id = f"agg_sub1_solution_path_{proposal.review_count:03d}" + + async def call(messages): + return await api_client_manager.generate_completion( + task_id=task_id, + role_id=reviewer_role_id, + model=primary.model_id, + messages=messages, + max_tokens=primary.max_output_tokens, + temperature=0.0, + ) + + return await review_with_json_retry( + prompt=prompt, + call_completion=call, + extract_text=lambda response: extract_message_text( + response["choices"][0]["message"] + ), + context_window=primary.context_window, + max_output_tokens=primary.max_output_tokens, + compact_prompt=compact_review_prompt( + user_prompt=self._base_user_research_prompt, + proposal=proposal, + current_plan=current_plan, + ), + ) + + run_id = getattr(session_manager, "session_id", "") or "autonomous" + root = Path(system_config.data_dir) / "solution_paths" + self.solution_path_manager = await solution_path_registry.acquire( + root, + workflow_mode="autonomous", + user_prompt=self._base_user_research_prompt, + stable_run_id=run_id, + reviewer=reviewer, + ) + cumulative = max( + self._solution_path_acceptance_count, + self.solution_path_manager.state.acceptance_count, + ) + self._solution_path_acceptance_count = cumulative + await self.solution_path_manager.set_acceptance_count(cumulative) + async def start(self) -> None: """Start the autonomous research loop.""" if self._running: @@ -2465,57 +2672,49 @@ async def start(self) -> None: self._running = True self._stop_event.clear() self._state.is_running = True + api_client_manager.set_assistant_memory_suppressed( + "autonomous_allowed_outputs", + not self._allow_mathematical_proofs, + ) self._stop_broadcast_sent = False self._fatal_stop_reason = None self._fatal_stop_message = "" - - # Reset free model manager state for fresh start - free_model_manager.reset() - - # Reset free model manager state for fresh start - free_model_manager.reset() - - # Set up autonomous API logging callback - async def log_callback(task_id, role_id, model, provider, prompt, response, - tokens_used, duration_ms, success, error, phase): - """Callback for logging autonomous API calls.""" - try: - await autonomous_api_logger.log_api_call( - task_id=task_id, - role_id=role_id, - model=model, - provider=provider, - prompt=prompt, - response_content=response, - tokens_used=tokens_used, - duration_ms=duration_ms, - success=success, - error=error, - phase=phase - ) - except Exception as e: - logger.error(f"Failed to log API call in autonomous logger: {e}") - - api_client_manager.set_autonomous_logger_callback(log_callback) - logger.info("Autonomous API logging enabled") - - # Reset and start token tracking for this session - token_tracker.reset() - token_tracker.start_timer() - - # Refresh workflow predictions at start - await self.refresh_workflow_predictions() - - await self._broadcast("auto_research_started") - logger.info("AutonomousCoordinator started") - - # Check for interrupted workflow to resume - resume_state = await self._get_resume_point() - - if not resume_state: - await self._run_proof_framing_gate() - + self._fatal_stop_payload = {} try: + free_model_manager.reset() + + async def log_callback(task_id, role_id, model, provider, prompt, response, + tokens_used, duration_ms, success, error, phase): + """Callback for logging autonomous API calls.""" + try: + await autonomous_api_logger.log_api_call( + task_id=task_id, + role_id=role_id, + model=model, + provider=provider, + prompt=prompt, + response_content=response, + tokens_used=tokens_used, + duration_ms=duration_ms, + success=success, + error=error, + phase=phase + ) + except Exception as e: + logger.error(f"Failed to log API call in autonomous logger: {e}") + + api_client_manager.set_autonomous_logger_callback(log_callback) + logger.info("Autonomous API logging enabled") + token_tracker.reset() + token_tracker.start_timer() + await self.refresh_workflow_predictions() + await self._broadcast("auto_research_started") + logger.info("AutonomousCoordinator started") + + resume_state = await self._get_resume_point() + if not resume_state: + await self._run_proof_framing_gate() + # Main research loop while self._running and not self._stop_event.is_set(): try: @@ -3202,10 +3401,27 @@ async def log_callback(task_id, role_id, model, provider, prompt, response, await self._save_workflow_state() raise finally: + api_client_manager.set_assistant_memory_suppressed( + "autonomous_allowed_outputs", + False, + ) self._running = False self._state.is_running = False + if self.solution_path_manager is not None: + try: + await self.solution_path_manager.stop() + except Exception: + logger.exception( + "Failed to stop Autonomous solution-path worker during terminal cleanup" + ) token_tracker.stop_timer() await self._broadcast_stopped_once() + if not self._running and not self._ownership_handoff_active: + if self.top_level_terminal_callback is not None: + try: + self.top_level_terminal_callback() + except Exception: + logger.exception("Autonomous top-level terminal callback failed") logger.info("AutonomousCoordinator stopped") async def _get_resume_point(self) -> Optional[Dict[str, Any]]: @@ -3310,6 +3526,8 @@ async def stop(self) -> None: self._stop_event.set() self._running = False self._state.is_running = False + if self.solution_path_manager is not None: + await self.solution_path_manager.stop() await self._broadcast_stopped_once() async def _run_shutdown_step(label: str, awaitable, timeout: float = 5.0) -> bool: @@ -3336,7 +3554,7 @@ def _consume_shutdown_exception(done_task: asyncio.Task) -> None: return False # Stop any running aggregator or compiler to prevent orphan tasks - await self._stop_active_child_aggregators("autonomous stop") + await self._stop_active_child_aggregators("autonomous stop", timeout=5.0) if self._brainstorm_aggregator: try: @@ -3377,7 +3595,7 @@ def _consume_shutdown_exception(done_task: asyncio.Task) -> None: main_task.cancel() done, _ = await asyncio.wait({main_task}, timeout=5) if main_task not in done: - logger.warning("AutonomousCoordinator background task is still cancelling") + raise RuntimeError("AutonomousCoordinator background task did not stop") logger.info("Autonomous research stopped - press Start to resume from last state") @@ -3719,6 +3937,9 @@ async def _resume_research_loop_after_tier3(self) -> None: self._running = False self._state.is_running = False token_tracker.stop_timer() + from backend.api.routes.autonomous import _release_autonomous_workflow_lease + if not self._running and not self._ownership_handoff_active: + _release_autonomous_workflow_lease() shared_training_memory.insights.clear() shared_training_memory.submission_count = 0 @@ -3834,6 +4055,8 @@ async def _topic_exploration_phase(self) -> str: creativity_emphasis_boost_enabled=self._creativity_emphasis_boost_enabled, enable_cleanup_review=False, proof_database_store=proof_database, + solution_path_manager=self.solution_path_manager, + solution_path_acceptance_count_owner=False, local_rejection_log_dir=str(brainstorm_memory._base_dir), local_rejection_log_template="topic_exploration_submitter_{submitter_id}_rejections.txt", reset_local_rejection_logs_on_start=True, @@ -3858,7 +4081,9 @@ async def _topic_exploration_phase(self) -> str: "Topic exploration stopped for context overflow: %s", getattr(exploration_aggregator, "fatal_error_message", ""), ) - self._mark_context_overflow_stop() + self._mark_context_overflow_stop( + getattr(exploration_aggregator, "fatal_error_payload", None) + ) self._stop_event.set() return "" if not status.is_running: @@ -4104,28 +4329,6 @@ async def _execute_topic_selection( logger.info(f"Continuing brainstorm: {topic_id}") return topic_id - - elif submission.action == "combine_topics": - # Combine multiple brainstorms - topic_id = await research_metadata.generate_topic_id() - metadata = await brainstorm_memory.combine_topics( - new_topic_id=topic_id, - new_topic_prompt=submission.topic_prompt, - source_topic_ids=submission.topic_ids - ) - - if metadata is None: - logger.error("Failed to combine topics") - return None - - await research_metadata.register_brainstorm(metadata) - - self._current_topic_id = topic_id - self._acceptance_count = metadata.submission_count - self._consecutive_rejections = 0 - - logger.info(f"Combined topics into: {topic_id}") - return topic_id return None @@ -4199,6 +4402,14 @@ async def _brainstorm_continuation_decision(self) -> str: papers_written_count=self._brainstorm_paper_count, rejection_context=rejection_context ) + from backend.shared.solution_path.integration import with_budgeted_solver_plan + max_input = rag_config.get_available_input_tokens( + self._topic_selector.context_window, + self._topic_selector.max_output_tokens, + ) + prompt = with_budgeted_solver_plan( + prompt, self.solution_path_manager, max_input + ) task_id = f"auto_cd_{self._topic_selector.task_sequence:03d}" self._topic_selector.task_sequence += 1 @@ -4644,6 +4855,8 @@ async def hard_limit_callback(total_acceptances: int) -> None: allow_trusted_context_files=True, trusted_context_texts=reference_brainstorm_contexts, proof_database_store=proof_database, + solution_path_manager=self.solution_path_manager, + solution_path_acceptance_count_owner=False, local_rejection_log_dir=str(brainstorm_memory._base_dir), local_rejection_log_template=( f"brainstorm_{brainstorm_memory._safe_topic_id(self._current_topic_id)}" @@ -4766,7 +4979,9 @@ async def handle_manual_override() -> bool: "Brainstorm aggregation stopped for context overflow: %s", getattr(self._brainstorm_aggregator, "fatal_error_message", ""), ) - self._mark_context_overflow_stop() + self._mark_context_overflow_stop( + getattr(self._brainstorm_aggregator, "fatal_error_payload", None) + ) self._stop_event.set() return False current_acceptances = status.total_acceptances @@ -4809,6 +5024,11 @@ async def handle_manual_override() -> bool: if current_acceptances > last_acceptances: new_acceptances = current_acceptances - last_acceptances self._acceptance_count = resume_acceptance_base + current_acceptances + self._solution_path_acceptance_count += new_acceptances + if self.solution_path_manager is not None: + await self.solution_path_manager.set_acceptance_count( + self._solution_path_acceptance_count + ) self._consecutive_rejections = 0 last_acceptances = current_acceptances @@ -4956,9 +5176,9 @@ async def _check_early_completion_triggers(self) -> bool: # Check for exhaustion signals in recent rejection logs # Parse last 5 rejections from each submitter for exhaustion keywords exhaustion_keywords = [ - "cannot identify new mathematical content", + "cannot identify new high-impact content", "brainstorm topic appears thoroughly explored", - "all major mathematical avenues have been covered", + "all major solution avenues have been covered", "exhausted", "no new insights" ] @@ -5116,6 +5336,7 @@ async def force_tier3_final_answer(self, mode: str = "complete_current") -> dict # where the main loop continues creating new brainstorms while Tier 3 runs self._running = False self._stop_event.set() + self._ownership_handoff_active = True logger.info("Force Tier 3: Main loop stopped") # Stop current aggregator if it exists (don't check tier - state is unreliable) @@ -5187,6 +5408,9 @@ async def force_tier3_final_answer(self, mode: str = "complete_current") -> dict # Reset flags to stopped state - research is complete self._running = False self._stop_event.set() + self._ownership_handoff_active = False + from backend.api.routes.autonomous import _release_autonomous_workflow_lease + _release_autonomous_workflow_lease() logger.info("Force Tier 3: Flags reset to stopped state after successful completion") try: @@ -5212,6 +5436,7 @@ async def force_tier3_final_answer(self, mode: str = "complete_current") -> dict # Create a tracked background task to resume the main research loop. self._main_task = asyncio.create_task(self._resume_research_loop_after_tier3()) self._main_task.add_done_callback(self._on_main_task_done) + self._ownership_handoff_active = False return { "success": True, @@ -5222,6 +5447,10 @@ async def force_tier3_final_answer(self, mode: str = "complete_current") -> dict return {"success": False, "result": "error", "message": "Invalid mode or state"} except Exception as e: + self._ownership_handoff_active = False + if not self.is_active: + from backend.api.routes.autonomous import _release_autonomous_workflow_lease + _release_autonomous_workflow_lease() logger.error(f"Error forcing Tier 3: {e}", exc_info=True) return {"success": False, "result": "error", "message": "An internal error occurred during Tier 3 processing"} @@ -5722,6 +5951,8 @@ async def _paper_title_exploration_phase( creativity_emphasis_boost_enabled=self._creativity_emphasis_boost_enabled, enable_cleanup_review=False, proof_database_store=proof_database, + solution_path_manager=self.solution_path_manager, + solution_path_acceptance_count_owner=False, local_rejection_log_dir=str(brainstorm_memory._base_dir), local_rejection_log_template=( f"title_candidates_{re.sub(r'[^A-Za-z0-9_-]+', '_', topic_suffix)}" @@ -5751,7 +5982,9 @@ async def _paper_title_exploration_phase( "Paper title exploration stopped for context overflow: %s", getattr(exploration_aggregator, "fatal_error_message", ""), ) - self._mark_context_overflow_stop() + self._mark_context_overflow_stop( + getattr(exploration_aggregator, "fatal_error_payload", None) + ) self._stop_event.set() return "" if not status.is_running: @@ -5904,13 +6137,14 @@ async def _compile_paper( # Initialize compiler for this paper self._paper_compiler = CompilerCoordinator() + self._paper_compiler.solution_path_manager = self.solution_path_manager try: # CRITICAL: Clear RAG before autonomous paper compilation to prevent cross-contamination # This removes any old user uploads, previous session data, or Part 1 aggregator content # Even on resume, we need to reload RAG since it's not persisted across restarts logger.info("Clearing RAG for autonomous paper compilation...") - await asyncio.to_thread(rag_manager.clear_all_documents) + await rag_manager.clear_all_documents_async() logger.info("RAG cleared successfully") # Initialize compiler with paper title as prompt @@ -5995,7 +6229,7 @@ async def _compile_paper( # Load brainstorm database into compiler RAG # This is now the ONLY aggregator content loaded (no Part 1 pollution) # Proof sections (both novel and non-novel) are stripped before indexing - # so that RAG chunks contain only mathematical submission content. + # so RAG chunks contain only ordinary brainstorm submission content. # Novel proofs reach the compiler via proof_database.inject_into_prompt(). brainstorm_db_path = brainstorm_memory.get_database_path(self._current_topic_id) if os.path.exists(brainstorm_db_path): @@ -6116,7 +6350,9 @@ async def _compile_paper( "Paper compiler stopped for context overflow: %s", getattr(self._paper_compiler, "fatal_error_message", ""), ) - self._mark_context_overflow_stop() + self._mark_context_overflow_stop( + getattr(self._paper_compiler, "fatal_error_payload", None) + ) self._stop_event.set() logger.warning("Compiler stopped unexpectedly") break @@ -6509,7 +6745,9 @@ async def _run_completed_paper_proof_checks( self._stop_event.set() return "stopped" if self._stop_event.is_set() else retry_status - await research_metadata.clear_proof_checkpoint("paper", paper_id) + proof_checkpoint = await research_metadata.get_proof_checkpoint("paper", paper_id) + if not proof_checkpoint or proof_checkpoint.get("status") != "deferred": + await research_metadata.clear_proof_checkpoint("paper", paper_id) return "complete" async def _auto_generate_paper_critique( @@ -6551,6 +6789,8 @@ async def _auto_generate_paper_critique( paper_title=paper_title, custom_prompt=None # Use default prompt ) + from backend.shared.solution_path.integration import with_validator_hook + prompt = with_validator_hook(prompt, self.solution_path_manager) # Wrap in messages list for API call messages = [{"role": "user", "content": prompt}] @@ -6584,8 +6824,11 @@ async def _auto_generate_paper_critique( ) # Generate critique + critique_task_id = ( + f"auto_paper_critique_{paper_id}_{datetime.now().strftime('%Y%m%d_%H%M%S')}" + ) response = await api_client_manager.generate_completion( - task_id=f"auto_paper_critique_{paper_id}_{datetime.now().strftime('%Y%m%d_%H%M%S')}", + task_id=critique_task_id, role_id="paper_critic", model=self._validator_model, messages=messages, @@ -6606,6 +6849,15 @@ async def _auto_generate_paper_critique( # Parse JSON with lenient fallback for truncated responses from backend.shared.critique_prompts import parse_critique_response critique_data = parse_critique_response(response_content) + from backend.shared.solution_path.integration import enqueue_optional_update + await enqueue_optional_update( + critique_data, + self.solution_path_manager, + proposer_role="paper_critic", + source_task_id=critique_task_id, + source_phase="completed_paper_auto_critique", + source_decision="accept", + ) # Extract ratings novelty = critique_data.get("novelty_rating", 0) @@ -7084,7 +7336,7 @@ async def _resume_tier3_long_form(self, tier3_state) -> bool: # Continue writing remaining chapters for chapter in remaining_chapters: if self._stop_event.is_set(): - break + return False await final_answer_memory.set_current_writing_chapter(chapter.order) @@ -7110,6 +7362,9 @@ async def _resume_tier3_long_form(self, tier3_state) -> bool: "chapter_order": chapter.order, "title": chapter.title }) + + if self._stop_event.is_set(): + return False # Assemble final volume await final_answer_memory.assemble_final_volume() @@ -7437,7 +7692,7 @@ async def _tier3_long_form_workflow( for chapter in chapters_to_write: if self._stop_event.is_set(): - break + return False await final_answer_memory.set_current_writing_chapter(chapter.order) @@ -7462,6 +7717,9 @@ async def _tier3_long_form_workflow( "chapter_order": chapter.order, "title": chapter.title }) + + if self._stop_event.is_set(): + return False # Step 3: Assemble final volume await final_answer_memory.assemble_final_volume() @@ -7522,7 +7780,10 @@ async def _tier3_title_selection( # Run title exploration phase for Tier 3 topic_prompt = f"[TIER 3 FINAL ANSWER] Certainty: {assessment.certainty_level}" - brainstorm_summary = f"Known Certainties:\n{assessment.known_certainties_summary}" + brainstorm_summary = ( + "Defensible Findings and Evidence Status:\n" + f"{assessment.known_certainties_summary}" + ) candidate_titles = await self._paper_title_exploration_phase( topic_prompt=topic_prompt, @@ -7556,7 +7817,9 @@ async def _compile_tier3_paper( paper_id: str, paper_title: str, reference_paper_ids: List[str], - assessment + assessment, + writing_context: str = "", + generated_chapter_context: str = "", ) -> Optional[str]: """ Compile Tier 3 final answer paper. @@ -7577,18 +7840,19 @@ async def _compile_tier3_paper( # Initialize compiler for this paper self._paper_compiler = CompilerCoordinator() + self._paper_compiler.solution_path_manager = self.solution_path_manager try: # Clear RAG for fresh Tier 3 compilation logger.info("Clearing RAG for Tier 3 paper compilation...") - await asyncio.to_thread(rag_manager.clear_all_documents) + await rag_manager.clear_all_documents_async() # Initialize compiler await self._paper_compiler.initialize( - compiler_prompt=self._apply_proof_context( - f"Write a mathematical research paper titled: {paper_title}\n\n" - f"IMPORTANT: This paper directly answers the research question.\n" - f"Known Certainties: {assessment.known_certainties_summary}" + compiler_prompt=self._build_tier3_compiler_prompt( + paper_title=paper_title, + findings_summary=assessment.known_certainties_summary, + writing_context=writing_context, ), validator_model=self._validator_model, writer_model=self._writer_model, @@ -7648,6 +7912,14 @@ async def _compile_tier3_paper( logger.info(f"Tier 3 reference loaded with proof sections stripped: {ref_paper_id}") else: logger.warning(f"Tier 3 reference paper was empty after proof stripping: {ref_paper_id}") + + if generated_chapter_context.strip(): + await rag_manager.add_text( + generated_chapter_context, + f"tier3_generated_chapter_context_{paper_id}.txt", + chunk_sizes=[512], + is_permanent=True, + ) # Start compiler await self._paper_compiler.start() @@ -7668,7 +7940,9 @@ async def _compile_tier3_paper( "Tier 3 compiler stopped for context overflow: %s", getattr(self._paper_compiler, "fatal_error_message", ""), ) - self._mark_context_overflow_stop() + self._mark_context_overflow_stop( + getattr(self._paper_compiler, "fatal_error_payload", None) + ) self._stop_event.set() break @@ -7709,11 +7983,27 @@ async def _write_volume_chapter( # Determine context based on chapter type if chapter.chapter_type == "introduction": - context = "Write the INTRODUCTION for this volume. You have access to ALL chapters." + context = ( + "Write the INTRODUCTION for this volume. Frame the direct solution, " + "its dependency structure, and the evidence status of each component " + "using the available existing-paper chapter references. " + "Use domain-appropriate rigor and retain mathematical foundations or " + "formal proofs when relevant." + ) elif chapter.chapter_type == "conclusion": - context = "Write the CONCLUSION for this volume. Synthesize findings from all body chapters." + context = ( + "Write the CONCLUSION for this volume. Synthesize the strongest " + "defensible direct answer from the available existing-paper chapter " + "references, distinguishing " + "demonstrated results from proposals and required validation. Use " + "domain-appropriate rigor and retain mathematical results when relevant." + ) else: - context = f"Write a paper to fill this content gap: {chapter.description}" + context = ( + "Write a rigorous direct-solution paper that closes this necessary " + "proof, design, implementation, evidence, evaluation, validation, " + f"safety, risk, or other solution gap: {chapter.description}" + ) # Get reference papers (existing papers in the volume) reference_ids = [ @@ -7726,7 +8016,10 @@ async def _write_volume_chapter( candidate_titles = await self._paper_title_exploration_phase( topic_prompt=f"[VOLUME CHAPTER: {chapter.chapter_type}] {context}", - brainstorm_summary=f"Known Certainties:\n{assessment.known_certainties_summary}", + brainstorm_summary=( + "Defensible Findings and Evidence Status:\n" + f"{assessment.known_certainties_summary}" + ), existing_papers=[], reference_papers=ref_details ) @@ -7742,7 +8035,10 @@ async def _write_volume_chapter( chapter_title = await self._title_selector.select_title( user_research_prompt=self._get_effective_user_research_prompt(), topic_prompt=f"[VOLUME CHAPTER: {chapter.chapter_type}] {context}", - brainstorm_summary=f"Known Certainties:\n{assessment.known_certainties_summary}", + brainstorm_summary=( + "Defensible Findings and Evidence Status:\n" + f"{assessment.known_certainties_summary}" + ), existing_papers_from_brainstorm=[], reference_papers=ref_details, candidate_titles=candidate_titles, @@ -7751,15 +8047,51 @@ async def _write_volume_chapter( if chapter_title: chapter.title = chapter_title + else: + logger.warning( + "Volume chapter final title selection failed; preserving chapter for retry" + ) + return False # Compile the chapter paper chapter_paper_id = f"volume_ch{chapter.order:02d}_{chapter.chapter_type}" + + generated_context_parts = [] + for prior_chapter in sorted(volume.chapters, key=lambda item: item.order): + if prior_chapter.order == chapter.order: + continue + if prior_chapter.chapter_type not in { + "gap_paper", + "conclusion", + "introduction", + }: + continue + prior_content = await final_answer_memory.get_chapter_paper( + prior_chapter.order + ) + if prior_content.strip(): + generated_context_parts.append( + f"GENERATED VOLUME CHAPTER {prior_chapter.order} " + f"({prior_chapter.chapter_type}) — {prior_chapter.title}\n" + f"{prior_content}" + ) + generated_chapter_context = "\n\n".join(generated_context_parts) paper_content = await self._compile_tier3_paper( paper_id=chapter_paper_id, paper_title=chapter.title, reference_paper_ids=reference_ids, - assessment=assessment + assessment=assessment, + writing_context=( + f"Chapter type: {chapter.chapter_type}. {context} " + + ( + f"Organized chapter requirement: {chapter.description}" + if chapter.description + and chapter.description not in context + else "" + ) + ), + generated_chapter_context=generated_chapter_context, ) if paper_content: @@ -7787,11 +8119,21 @@ async def clear_all_data(self) -> None: # Check both internal flag and state object if self._running or self._state.is_running: raise RuntimeError("Cannot clear data while running") + if self.solution_path_manager is not None: + from backend.shared.solution_path import solution_path_registry + await solution_path_registry.clear_manager(self.solution_path_manager) + self.solution_path_manager = None + else: + from backend.shared.solution_path import solution_path_registry + run_id = getattr(session_manager, "session_id", "") or "" + if run_id: + await solution_path_registry.clear_run( + Path(system_config.data_dir) / "solution_paths", run_id + ) import json import shutil import time - from pathlib import Path from backend.aggregator.core.queue_manager import queue_manager # Wait briefly for any pending async file operations to complete @@ -7880,6 +8222,19 @@ def safe_rmtree(path: Path, max_retries: int = 5) -> bool: except Exception as e: # Non-critical: workflow state files are small logger.warning(f"Could not clear workflow state for {session_dir.name}: {e}") + + try: + safe_session_name = validate_single_path_component( + session_dir.name, "session ID" + ) + failed_proof_hints_dir = resolve_path_within_root( + sessions_dir, safe_session_name, "proofs", "failed" + ) + if failed_proof_hints_dir.exists(): + safe_rmtree(failed_proof_hints_dir) + logger.info(f"Cleared failed proof retry hints from session: {session_dir.name}") + except Exception as e: + logger.warning(f"Could not clear failed proof retry hints for {session_dir.name}: {e}") if session_mark_failures: critical_errors.append( "Failed to mark one or more sessions non-resumable: " @@ -7902,6 +8257,7 @@ def safe_rmtree(path: Path, max_retries: int = 5) -> bool: research_metadata.set_session_manager(None) final_answer_memory.set_session_manager(None) proof_database.set_session_manager(None) + await proof_database.clear_failed_candidates() successes.append("Reset live session path bindings") logger.info("Reset live session path bindings after clear") except Exception as e: @@ -7979,7 +8335,7 @@ def safe_rmtree(path: Path, max_retries: int = 5) -> bool: await asyncio.sleep(0.5) autonomous_rag_manager.reset() - await asyncio.to_thread(rag_manager.clear_all_documents) + await rag_manager.clear_all_documents_async() successes.append("Cleared RAG state") logger.info("Cleared RAG state (ChromaDB collections)") except Exception as e: @@ -8003,6 +8359,7 @@ def safe_rmtree(path: Path, max_retries: int = 5) -> bool: self._current_reference_papers = [] self._current_reference_brainstorms = [] self._acceptance_count = 0 + self._solution_path_acceptance_count = 0 self._rejection_count = 0 self._cleanup_removals = 0 self._consecutive_rejections = 0 diff --git a/backend/autonomous/core/autonomous_rag_manager.py b/backend/autonomous/core/autonomous_rag_manager.py index ea70c2a..4fdbde2 100644 --- a/backend/autonomous/core/autonomous_rag_manager.py +++ b/backend/autonomous/core/autonomous_rag_manager.py @@ -273,7 +273,7 @@ async def get_reference_papers_context( # Retrieve relevant chunks via RAG if not query: - query = "reference paper content mathematical research" + query = "reference paper content relevant to the exact objective, solution mechanism, evidence, constraints, and validation" # Get outline for context enhancement (use first paper's outline) outline = None @@ -424,7 +424,7 @@ async def prepare_compiler_context( } # RAG query for retrievals - rag_query = query or f"mathematical research paper compilation" + rag_query = query or "research paper or solution report compilation relevant to the exact objective" rag_exclude_sources: List[str] = [] # Priority 1: Brainstorm database (highest priority after outline) diff --git a/backend/autonomous/core/proof_novelty.py b/backend/autonomous/core/proof_novelty.py index 7495014..c6257aa 100644 --- a/backend/autonomous/core/proof_novelty.py +++ b/backend/autonomous/core/proof_novelty.py @@ -12,12 +12,17 @@ from __future__ import annotations import logging -from typing import Tuple +from typing import Any, Tuple from backend.autonomous.prompts.proof_prompts import build_proof_novelty_prompt -from backend.shared.api_client_manager import api_client_manager +from backend.shared.api_client_manager import RetryableProviderError, api_client_manager from backend.shared.config import rag_config -from backend.shared.json_parser import parse_json +from backend.shared.json_parser import parse_json, sanitize_model_output_for_retry_context +from backend.shared.model_error_utils import ( + is_non_retryable_model_error, + is_transient_model_call_error, +) +from backend.shared.openrouter_client import FreeModelExhaustedError from backend.shared.response_extraction import extract_message_text from backend.shared.utils import count_tokens @@ -35,6 +40,96 @@ ) +def _extract_novelty_content(response: dict[str, Any], label: str) -> str: + if not response or not response.get("choices"): + raise ValueError(f"Novelty validator {label} returned no response.") + message = response["choices"][0].get("message", {}) + content = extract_message_text(message) + if not content: + raise ValueError(f"Novelty validator {label} returned empty content.") + return content + + +def _parse_novelty_payload(content: str) -> Tuple[str, str]: + data = parse_json(content) + if isinstance(data, list): + if not data: + raise ValueError("Novelty validator returned an empty JSON array.") + data = data[0] + if not isinstance(data, dict): + raise ValueError("Novelty validator JSON was not an object.") + + raw_tier = str(data.get("novelty_tier", "")).strip().lower() + if not raw_tier: + raise ValueError("Novelty validator JSON omitted novelty_tier.") + if raw_tier not in VALID_NOVELTY_TIERS: + raise ValueError(f"Novelty validator returned unrecognised tier {raw_tier!r}.") + + return raw_tier, str(data.get("reasoning", "")).strip() + + +async def _retry_novelty_payload( + *, + prompt: str, + task_id: str, + role_id: str, + validator_model: str, + validator_context: int, + validator_max_tokens: int, + failed_output: str, + error: Exception, +) -> Tuple[str, str]: + logger.info("Novelty validator output failed; attempting bounded JSON retry: %s", error) + + retry_prompt = ( + "Your previous proof-novelty response could not be used.\n\n" + f"ERROR: {error}\n\n" + "Return the same novelty decision in valid JSON only. " + "Use this exact top-level shape:\n" + "{\n" + ' "novelty_tier": "not_novel | novel_formulation | ' + 'novel_variant | mathematical_discovery | major_mathematical_discovery",\n' + ' "reasoning": "brief explanation"\n' + "}\n\n" + "Respond with ONLY the JSON object, no markdown and no explanation." + ) + + max_input_tokens = rag_config.get_available_input_tokens(validator_context, validator_max_tokens) + failed_output_preview = sanitize_model_output_for_retry_context( + failed_output, + max_chars=2000, + ) + retry_messages = [ + {"role": "user", "content": prompt}, + {"role": "assistant", "content": failed_output_preview}, + {"role": "user", "content": retry_prompt}, + ] + if sum(count_tokens(str(message.get("content") or "")) for message in retry_messages) > max_input_tokens: + retry_messages[1]["content"] = ( + "[failed output omitted because retry context would exceed the model input budget]" + ) + if sum(count_tokens(str(message.get("content") or "")) for message in retry_messages) > max_input_tokens: + prompt_with_retry_instruction = f"{prompt}\n\n---\n{retry_prompt}" + if count_tokens(prompt_with_retry_instruction) <= max_input_tokens: + retry_messages = [{"role": "user", "content": prompt_with_retry_instruction}] + else: + logger.warning( + "Novelty validator retry instruction too large; retrying original prompt." + ) + retry_messages = [{"role": "user", "content": prompt}] + + retry_response = await api_client_manager.generate_completion( + task_id=f"{task_id}_retry", + role_id=role_id, + model=validator_model, + messages=retry_messages, + max_tokens=validator_max_tokens, + temperature=0.0, + ) + retry_content = _extract_novelty_content(retry_response, "retry") + return _parse_novelty_payload(retry_content) + + async def assess_proof_novelty( *, user_prompt: str, @@ -47,7 +142,7 @@ async def assess_proof_novelty( task_id: str, role_id: str = "autonomous_proof_novelty", ) -> Tuple[str, str]: - """Classify a Lean-4-verified theorem into one of five novelty tiers. + """Classify a Lean-4-verified theorem into one of the shared novelty tiers. Args: user_prompt: Top-level research prompt for context. @@ -56,8 +151,9 @@ async def assess_proof_novelty( validator_model: Model identifier to drive the novelty judgement. validator_context: Validator model's context window. validator_max_tokens: Maximum output tokens reserved for the judgement. - existing_novel_proofs: Pre-formatted block listing already-novel proofs; - trimmed in-loop if it overflows the validator budget. + existing_novel_proofs: Compatibility parameter. Shared registration + passes an empty string so private proof history cannot influence + the independent novelty judgment. task_id: Caller-chosen task id used for workflow tracking. role_id: Role identifier forwarded to the API client manager. Defaults to the autonomous role; the compiler rigor caller passes a @@ -67,8 +163,8 @@ async def assess_proof_novelty( Tuple of (novelty_tier, reasoning) where novelty_tier is one of: "not_novel", "novel_formulation", "novel_variant", "mathematical_discovery", "major_mathematical_discovery". - Falls back to ("not_novel", ) when the validator returns no - usable response or an unrecognised tier string. + Falls back to ("not_novel", ) only after the bounded JSON retry + path cannot produce a usable novelty decision. """ prompt = build_proof_novelty_prompt( user_prompt=user_prompt, @@ -99,28 +195,36 @@ async def assess_proof_novelty( max_tokens=validator_max_tokens, temperature=0.0, ) - if not response or not response.get("choices"): - return "not_novel", "Novelty validator returned no response." - - message = response["choices"][0].get("message", {}) - content = extract_message_text(message) - if not content: - return "not_novel", "Novelty validator returned empty content." + content = "" try: - data = parse_json(content) + content = _extract_novelty_content(response, "initial") + return _parse_novelty_payload(content) except Exception as exc: - logger.warning("Novelty validator JSON parse failed: %s", exc) - return "not_novel", f"Novelty validator JSON parse error: {exc}" - - if isinstance(data, list): - data = data[0] if data else {} - - raw_tier = str(data.get("novelty_tier", "not_novel")).strip().lower() - if raw_tier not in VALID_NOVELTY_TIERS: - logger.warning( - "Novelty validator returned unrecognised tier %r; falling back to not_novel", raw_tier - ) - raw_tier = "not_novel" - - return raw_tier, str(data.get("reasoning", "")).strip() + try: + return await _retry_novelty_payload( + prompt=prompt, + task_id=task_id, + role_id=role_id, + validator_model=validator_model, + validator_context=validator_context, + validator_max_tokens=validator_max_tokens, + failed_output=content, + error=exc, + ) + except FreeModelExhaustedError: + raise + except RetryableProviderError: + raise + except Exception as retry_exc: + if is_non_retryable_model_error(retry_exc) or is_transient_model_call_error(retry_exc): + raise + logger.warning( + "Novelty validator JSON retry failed; falling back to not_novel: %s", + retry_exc, + ) + return ( + "not_novel", + "Novelty validator JSON retry failed after retry exhaustion: " + f"initial error: {exc}; retry error: {retry_exc}", + ) diff --git a/backend/autonomous/core/proof_registration.py b/backend/autonomous/core/proof_registration.py index 591173d..eab6041 100644 --- a/backend/autonomous/core/proof_registration.py +++ b/backend/autonomous/core/proof_registration.py @@ -12,7 +12,17 @@ from typing import Any, Awaitable, Callable, Optional from backend.autonomous.core.proof_novelty import assess_proof_novelty -from backend.shared.models import ProofAttemptFeedback, ProofDependency, ProofRecord +from backend.autonomous.memory.proof_database import is_prompt_injection_novel_tier +from backend.shared.models import ( + ProofArtifactPurpose, + ProofAttemptFeedback, + ProofDependency, + ProofRecord, +) +from backend.shared.proof_identity import ( + CANONICAL_PROOF_IDENTITY_VERSION, + canonical_proof_identity, +) logger = logging.getLogger(__name__) @@ -27,34 +37,96 @@ class RegisteredProof: duplicate: bool = False -def _normalize_for_duplicate_check(value: str) -> str: - return "\n".join((value or "").strip().splitlines()) - - async def _find_existing_proof( proof_database, *, source_type: Optional[str] = None, source_id: Optional[str] = None, + current_run_id: str = "", theorem_statement: str, lean_code: str, ) -> Optional[ProofRecord]: - """Return an existing proof for the theorem/code, optionally source-scoped.""" - normalized_statement = " ".join((theorem_statement or "").split()) - normalized_code = _normalize_for_duplicate_check(lean_code) + """Return an existing canonical theorem/code match, optionally source-scoped.""" + identity = canonical_proof_identity(theorem_statement, lean_code) try: - for proof in await proof_database.get_all_proofs(): + candidates: list[ProofRecord] = list(await proof_database.get_all_proofs()) + if hasattr(proof_database, "list_proof_library"): + for item in await proof_database.list_proof_library(novel_only=False): + if isinstance(item, dict): + try: + candidates.append(ProofRecord.model_validate(item)) + except Exception: + continue + base_dir = getattr(proof_database, "_base_dir", None) + if base_dir is not None and hasattr(proof_database, "list_proof_library_from_history"): + history_root = base_dir.parent / "manual_proof_runs" + for item in await proof_database.list_proof_library_from_history( + history_root, + novel_only=False, + ): + if isinstance(item, dict): + try: + candidates.append(ProofRecord.model_validate(item)) + except Exception: + continue + + seen: set[tuple[str, str]] = set() + for proof in candidates: + occurrence_key = (proof.run_id, proof.proof_id) + if occurrence_key in seen: + continue + seen.add(occurrence_key) if source_type is not None and proof.source_type != source_type: continue if source_id is not None and proof.source_id != source_id: continue - if " ".join((proof.theorem_statement or "").split()) != normalized_statement: + proof_identity = canonical_proof_identity( + proof.theorem_statement, + proof.lean_code, + ) + if proof_identity.key != identity.key: continue - if _normalize_for_duplicate_check(proof.lean_code) != normalized_code: + proof_run_id = str(proof.run_id or f"{proof.source_type}:{proof.source_id}") + if current_run_id and proof_run_id == current_run_id: continue return proof except Exception as exc: logger.debug("Existing proof lookup failed for %s %s: %s", source_type, source_id, exc) + try: + from backend.shared.proof_search.models import default_proof_search_corpora + from backend.shared.proof_search.search_service import proof_search_service + + enabled_internal_corpora = [ + corpus + for corpus in default_proof_search_corpora() + if corpus in {"moto", "manual", "leanoj"} + ] + matches = await proof_search_service.exact_identity_neighborhood( + theorem_statement_hashes=[identity.theorem_statement_hash], + lean_code_hashes=[identity.lean_code_hash], + corpora=enabled_internal_corpora, + exclude_run_ids=[current_run_id] if current_run_id else None, + identity_version=CANONICAL_PROOF_IDENTITY_VERSION, + limit=1, + ) + if matches: + match = matches[0] + return ProofRecord( + proof_id=match.proof_id, + theorem_statement=match.theorem_statement, + theorem_name=match.theorem_name, + formal_sketch=match.formal_sketch, + source_type=match.source_type, + source_id=match.source_id, + source_title=match.source_title, + run_id=match.run_id or match.session_id, + lean_code=match.lean_code, + novel=match.novelty_tier != "not_novel", + novelty_tier=match.novelty_tier or "not_novel", + novelty_reasoning=match.novelty_reasoning, + ) + except Exception as exc: + logger.debug("Archived proof identity lookup failed: %s", exc) return None @@ -82,7 +154,7 @@ async def _broadcast_registered_proof( if proof_label: event_payload["proof_label"] = proof_label - if record.novel: + if record.novel and is_prompt_injection_novel_tier(record.novelty_tier): await broadcast_fn("novel_proof_discovered", event_payload) else: await broadcast_fn("known_proof_verified", event_payload) @@ -139,56 +211,63 @@ async def register_verified_lean_proof( base_event: Optional[dict[str, Any]] = None, proof_label: str = "", retry_origin_source_id: str = "", + run_id: str = "", + artifact_purpose: ProofArtifactPurpose | None = None, + ownership_predicate: Optional[Callable[[], bool]] = None, ) -> RegisteredProof: """ Classify and store Lean-verified proof code using the shared novelty tiers. - Same-source duplicate detection is scoped to source type/id, theorem - statement, and Lean code. Cross-source exact theorem/code matches are - stored as non-novel reused proof occurrences without spending a novelty - API call. + Every Lean-verified occurrence is independently novelty-assessed and stored. + Historical theorem/code matches are evidence for the validator, not a + shortcut that can suppress or pre-classify the current-run occurrence. """ - existing = await _find_existing_proof( - proof_database, - source_type=source_type, - source_id=source_id, + novelty_tier, novelty_reasoning = await assess_proof_novelty( + user_prompt=user_prompt, theorem_statement=theorem_statement, lean_code=lean_code, + validator_model=validator_model, + validator_context=validator_context, + validator_max_tokens=validator_max_tokens, + existing_novel_proofs="", + task_id=task_id, + role_id=role_id, ) - if existing is not None: - await _broadcast_duplicate_proof( - broadcast_fn=broadcast_fn, - record=existing, - base_event=base_event, - proof_label=proof_label, - ) - return RegisteredProof(record=existing, duplicate=True) + independent_novelty_tier = novelty_tier + independent_novelty_reasoning = novelty_reasoning - global_existing = await _find_existing_proof( + active_session_id = getattr( + getattr(proof_database, "_session_manager", None), + "session_id", + None, + ) + prompt_run_id = run_id or active_session_id + if not prompt_run_id and hasattr(proof_database, "get_or_create_active_run_id"): + prompt_run_id = await proof_database.get_or_create_active_run_id() + resolved_run_id = prompt_run_id or f"{source_type}:{source_id}" + identity = canonical_proof_identity(theorem_statement, lean_code) + existing = await _find_existing_proof( proof_database, theorem_statement=theorem_statement, lean_code=lean_code, + current_run_id=resolved_run_id, ) - if global_existing is not None: - novelty_tier = "not_novel" - novelty_reasoning = ( - "Exact theorem statement and Lean code already exist in the proof database " - f"as {global_existing.proof_id}; this source records a reused verified proof occurrence." - ) - else: - existing_novel_proofs = proof_database.get_novel_proofs_for_injection() - novelty_tier, novelty_reasoning = await assess_proof_novelty( - user_prompt=user_prompt, - theorem_statement=theorem_statement, - lean_code=lean_code, - validator_model=validator_model, - validator_context=validator_context, - validator_max_tokens=validator_max_tokens, - existing_novel_proofs=existing_novel_proofs, - task_id=task_id, - role_id=role_id, + existing_run_id = ( + str(getattr(existing, "run_id", "") or "") + or ( + f"{getattr(existing, 'source_type', '')}:{getattr(existing, 'source_id', '')}" + if existing + else "" ) + ) + is_cross_run_duplicate = bool(existing and existing_run_id != resolved_run_id) + if is_cross_run_duplicate and novelty_tier != "not_novel": + novelty_tier = "duplicate_novel" + novelty_reasoning = independent_novelty_reasoning is_novel = novelty_tier != "not_novel" + resolved_artifact_purpose: ProofArtifactPurpose = ( + artifact_purpose or "verified_occurrence" + ) record = ProofRecord( proof_id="", @@ -199,18 +278,34 @@ async def register_verified_lean_proof( source_type=source_type, source_id=source_id, source_title=source_title, + run_id=resolved_run_id, + user_prompt=user_prompt, solver=solver, lean_code=lean_code, novel=is_novel, novelty_tier=novelty_tier, novelty_reasoning=novelty_reasoning, + independent_novelty_tier=independent_novelty_tier, + independent_novelty_reasoning=independent_novelty_reasoning, + exact_duplicate_proof_id=existing.proof_id if is_cross_run_duplicate else "", + exact_duplicate_run_id=existing_run_id if is_cross_run_duplicate else "", + artifact_purpose=resolved_artifact_purpose, + canonical_identity_version=identity.version, + canonical_theorem_statement_hash=identity.theorem_statement_hash, + canonical_lean_code_hash=identity.lean_code_hash, verification_notes=verification_notes, attempt_count=attempt_count, attempts=list(attempts or []), dependencies=list(dependencies or []), solver_hints=list(solver_hints or []), ) - if hasattr(proof_database, "add_proof_if_absent"): + if ownership_predicate is not None and not ownership_predicate(): + raise RuntimeError("Proof registration ownership was superseded") + + if hasattr(proof_database, "add_proof_occurrence"): + stored = await proof_database.add_proof_occurrence(record) + duplicate = False + elif hasattr(proof_database, "add_proof_if_absent"): stored, duplicate = await proof_database.add_proof_if_absent(record) else: stored = await proof_database.add_proof(record) diff --git a/backend/autonomous/core/proof_verification_stage.py b/backend/autonomous/core/proof_verification_stage.py index 2d001cf..27276d7 100644 --- a/backend/autonomous/core/proof_verification_stage.py +++ b/backend/autonomous/core/proof_verification_stage.py @@ -15,15 +15,21 @@ from backend.autonomous.agents.proof_identification_agent import ProofIdentificationAgent from backend.autonomous.memory.brainstorm_memory import brainstorm_memory from backend.autonomous.memory.paper_library import paper_library +from backend.autonomous.memory.proof_database import is_prompt_injection_novel_tier from backend.autonomous.core.proof_registration import register_verified_lean_proof from backend.shared.config import system_config +from backend.shared.context_overflow import ( + CONTEXT_OVERFLOW_RESOLUTION, + CONTEXT_OVERFLOW_STOP_REASON, + context_overflow_model_payload, +) from backend.shared.lean_proof_integrity import validate_full_lean_proof_integrity from backend.shared.model_error_utils import ( format_transient_provider_error, is_non_retryable_model_error, is_transient_model_call_error, ) -from backend.shared.api_client_manager import RetryableProviderError +from backend.shared.api_client_manager import RetryableProviderError, api_client_manager from backend.shared.models import ProofAttemptFeedback, ProofAttemptResult, ProofCandidate, ProofStageResult, SmtHint from backend.shared.openrouter_client import FreeModelExhaustedError from backend.shared.provider_pause import is_provider_credit_pause_error @@ -64,10 +70,21 @@ class ProofVerificationStage: _active_sources: set[str] = set() _active_sources_lock: Optional[asyncio.Lock] = None - def __init__(self) -> None: + def __init__(self, solution_path_manager: Any = None) -> None: self._novelty_task_sequence = 0 self._integrity_task_sequence = 0 self._dependency_extractor = ProofDependencyExtractor() + self.solution_path_manager = solution_path_manager + + @staticmethod + def _proof_workflow_mode(trigger: str) -> str: + if trigger == "manual_compiler_save": + return "compiler" + if trigger == "manual_compiler_aggregator": + return "aggregator" + if trigger == "manual": + return "manual_proof_check" + return "autonomous" @classmethod def _get_active_sources_lock(cls) -> asyncio.Lock: @@ -358,7 +375,11 @@ async def _run_smt_check( z3_output=z3_raw[:2000], ) except Exception as exc: - if is_non_retryable_model_error(exc): + if ( + is_non_retryable_model_error(exc) + or isinstance(exc, RetryableProviderError) + or is_transient_model_call_error(exc) + ): raise logger.debug("SMT check failed for theorem %s in %s %s: %s", candidate.theorem_id, source_type, source_id, exc) elapsed_ms = int((time.monotonic() - started_at) * 1000) @@ -461,6 +482,8 @@ async def run( proof_round_index: int = 1, proof_max_rounds: int = 1, prior_round_results: str = "", + canonical_user_prompt: str = "", + run_id: str = "", ) -> ProofStageResult: """Run proof identification, formalization, Lean 4 checking, and novelty review.""" result = ProofStageResult(source_type=source_type, source_id=source_id) @@ -501,6 +524,7 @@ async def save_checkpoint(status: str) -> None: for index, candidate in enumerate(list(resolved_candidates), start=1) ], "processed_candidate_ids": sorted(processed_candidate_ids), + "deferred_candidate_ids": list(result.deferred_candidate_ids), "attempts_by_candidate": { theorem_id: [ attempt.model_dump(mode="json") @@ -563,6 +587,7 @@ def _stop_requested() -> bool: context_window=submitter_context, max_output_tokens=submitter_max_tokens, role_id=f"autonomous_proof_identification_{role_suffix}", + solution_path_manager=self.solution_path_manager, ) resolved_candidates = await self._resolve_candidates( @@ -824,8 +849,14 @@ async def cancel_and_drain(extra_tasks=()) -> None: and ProofFormalizationAgent.is_context_overflow_feedback(attempts[-1]) ) if context_overflow: - result.had_error = True - result.error_message = error_summary + # This candidate is deferred, not failed. Do not add it + # to results or processed IDs: its checkpoint remains + # eligible after proof-model/context settings change. + if candidate.theorem_id not in result.deferred_candidate_ids: + result.deferred_candidate_ids.append(candidate.theorem_id) + mark_batch_outcome_processed(batch_index) + await save_checkpoint("running") + continue if source_type == "brainstorm" and trigger != "retry" and not context_overflow: await novel_proofs_db.record_failed_candidate( source_id, @@ -959,7 +990,7 @@ async def cancel_and_drain(extra_tasks=()) -> None: registration = await register_verified_lean_proof( proof_database=novel_proofs_db, - user_prompt=user_prompt, + user_prompt=canonical_user_prompt or user_prompt, theorem_statement=stored_theorem_statement, lean_code=lean_code, validator_model=validator_model, @@ -982,9 +1013,11 @@ async def cancel_and_drain(extra_tasks=()) -> None: base_event=base_event, proof_label=proof_label, retry_origin_source_id=candidate.origin_source_id, + run_id=run_id, ) stored_record = registration.record is_novel = stored_record.novel + is_prompt_novel = is_prompt_injection_novel_tier(stored_record.novelty_tier) result.verified_count += 1 await self._broadcast( @@ -1057,12 +1090,12 @@ async def cancel_and_drain(extra_tasks=()) -> None: ) if self._should_append_verified_proof( - is_novel=is_novel, + is_novel=is_prompt_novel, duplicate=registration.duplicate, append_proof_callback=append_proof_callback, append_known_proofs=self._should_append_known_proofs_for_trigger(trigger), ): - if is_novel and not registration.duplicate: + if is_prompt_novel and not registration.duplicate: result.novel_count += 1 if append_proof_callback is not None: await append_proof_callback(stored_record) @@ -1100,7 +1133,8 @@ async def cancel_and_drain(extra_tasks=()) -> None: if partial_stop: return result - await save_checkpoint("complete") + checkpoint_status = "deferred" if result.deferred_candidate_ids else "complete" + await save_checkpoint(checkpoint_status) await self._broadcast( broadcast_fn, "proof_check_complete", @@ -1109,6 +1143,7 @@ async def cancel_and_drain(extra_tasks=()) -> None: "novel_count": result.novel_count, "verified_count": result.verified_count, "total_candidates": result.total_candidates, + "deferred_candidate_ids": list(result.deferred_candidate_ids), }, ) return result @@ -1304,6 +1339,67 @@ async def on_attempt_feedback(feedback, current_candidate=candidate) -> None: "retry_origin_source_id": current_candidate.origin_source_id, }, ) + elif ProofFormalizationAgent.is_context_overflow_feedback(feedback): + configured_payload = context_overflow_model_payload( + api_client_manager.get_role_config(formalization_agent.role_id) + ) + route_payload = { + key: value + for key, value in { + "configured_model": feedback.configured_model, + "configured_provider": feedback.configured_provider, + "effective_model": feedback.effective_model, + "effective_provider": feedback.effective_provider, + }.items() + if value + } + overflow_origin = feedback.overflow_origin or "provider" + is_local_preflight = overflow_origin == "local_preflight" + message = ( + "Proof formalization deferred before provider invocation: the mandatory " + f"prompt requires {feedback.prompt_tokens:,} input tokens but the configured " + f"input budget is {feedback.max_input_tokens:,}. Lean 4 was not run. Increase " + "the proof model context window, reduce its output reserve, or reduce source context." + if is_local_preflight + and feedback.prompt_tokens is not None + and feedback.max_input_tokens is not None + else ( + "Proof formalization deferred before provider invocation because the mandatory " + "prompt exceeds the configured local input budget. Lean 4 was not run. Increase " + "the proof model context window, reduce its output reserve, or reduce source context." + if is_local_preflight + else ( + "Proof formalization deferred: the selected provider rejected the proof " + "prompt because it exceeded the model context window. Lean 4 was not run. " + "Choose a larger-context proof model or reduce source context." + ) + ) + ) + await self._broadcast( + broadcast_fn, + "proof_context_overflow", + { + **base_event, + "workflow_mode": self._proof_workflow_mode(trigger), + "fatal": False, + "overflow_origin": overflow_origin, + "role_id": formalization_agent.role_id, + **configured_payload, + **route_payload, + "reason": CONTEXT_OVERFLOW_STOP_REASON, + "message": message, + "resolution": CONTEXT_OVERFLOW_RESOLUTION, + "error_detail": feedback.error_output, + "theorem_id": current_candidate.theorem_id, + "theorem_statement": current_candidate.statement, + "proof_label": proof_label, + "attempt": feedback.attempt, + "strategy": feedback.strategy, + "prompt_tokens": feedback.prompt_tokens, + "max_input_tokens": feedback.max_input_tokens, + "retry_origin_source_id": current_candidate.origin_source_id, + }, + ) else: lean_response = self._lean_response_summary(feedback) await self._broadcast( @@ -1323,8 +1419,18 @@ async def on_attempt_feedback(feedback, current_candidate=candidate) -> None: }, ) - full_attempt_count = sum(1 for attempt in active_attempts if attempt.strategy == "full_script") - tactic_attempt_count = sum(1 for attempt in active_attempts if attempt.strategy == "tactic_script") + full_attempt_count = sum( + 1 + for attempt in active_attempts + if attempt.strategy == "full_script" + and not ProofFormalizationAgent.is_context_overflow_feedback(attempt) + ) + tactic_attempt_count = sum( + 1 + for attempt in active_attempts + if attempt.strategy == "tactic_script" + and not ProofFormalizationAgent.is_context_overflow_feedback(attempt) + ) full_remaining = max(0, 3 - full_attempt_count) tactic_remaining = max(0, 2 - tactic_attempt_count) success = False @@ -1412,6 +1518,8 @@ async def run_manual( source_type: str, source_id: str, user_prompt: str, + canonical_user_prompt: str = "", + run_id: str = "", submitter_model: str, submitter_context: int, submitter_max_tokens: int, @@ -1432,6 +1540,8 @@ async def run_manual( source_type=source_type, source_id=source_id, user_prompt=user_prompt, + canonical_user_prompt=canonical_user_prompt, + run_id=run_id, submitter_model=submitter_model, submitter_context=submitter_context, submitter_max_tokens=submitter_max_tokens, diff --git a/backend/autonomous/memory/autonomous_api_logger.py b/backend/autonomous/memory/autonomous_api_logger.py index fb2e649..c356d8d 100644 --- a/backend/autonomous/memory/autonomous_api_logger.py +++ b/backend/autonomous/memory/autonomous_api_logger.py @@ -7,12 +7,13 @@ import json import logging import os -from collections import deque +from collections import OrderedDict, deque from datetime import datetime from typing import Dict, Any, List, Optional from pathlib import Path from backend.shared.config import system_config +from backend.shared.api_log_persistence import atomic_write_log_lines, ensure_private_log_file from backend.shared.log_redaction import redact_log_text logger = logging.getLogger(__name__) @@ -51,17 +52,23 @@ def __init__(self): return self._initialized = True + self._prepared_root_identity = None + self._volatile_payloads: OrderedDict[str, Dict[str, str]] = OrderedDict() + logger.info("AutonomousAPILogger initialized") + + def _prepare_active_root(self) -> None: + identity = system_config.runtime_root_identity() + if self._prepared_root_identity == identity: + return + self._volatile_payloads.clear() self._ensure_log_file() self._scrub_persisted_full_payloads() - logger.info("AutonomousAPILogger initialized") + self._prepared_root_identity = identity def _ensure_log_file(self) -> None: """Ensure the log file and directory exist.""" log_path = self._get_log_path() - log_path.parent.mkdir(parents=True, exist_ok=True) - - if not log_path.exists(): - log_path.write_text("") + ensure_private_log_file(log_path) def _get_log_path(self) -> Path: """Return the instance-scoped autonomous API log path.""" @@ -119,8 +126,7 @@ def _scrub_persisted_full_payloads(self) -> None: scrubbed_lines.append(json.dumps(entry) + "\n") if changed: - with open(log_path, "w", encoding="utf-8") as f: - f.writelines(scrubbed_lines) + atomic_write_log_lines(log_path, scrubbed_lines) logger.info("Scrubbed legacy full prompt/response payloads from autonomous API log") except Exception as e: logger.warning(f"Failed to scrub legacy autonomous API log payloads: {e}") @@ -159,12 +165,21 @@ async def log_api_call( """ async with self._lock: try: + self._prepare_active_root() prompt_meta = _payload_metadata(prompt, 1000) response_meta = _payload_metadata(response_content, 2000) - store_full_payloads = bool(system_config.api_log_store_full_payloads) + store_full_payloads = bool( + system_config.api_log_store_full_payloads + and not system_config.generic_mode + ) + timestamp = datetime.now().isoformat() + entry_id = hashlib.sha256( + f"{timestamp}\0{task_id}\0{role_id}".encode("utf-8", errors="replace") + ).hexdigest() log_entry = { - "timestamp": datetime.now().isoformat(), + "entry_id": entry_id, + "timestamp": timestamp, "task_id": task_id, "role_id": role_id, "model": model, @@ -174,23 +189,29 @@ async def log_api_call( "prompt_preview": prompt_meta["preview"], "prompt_size": prompt_meta["size"], "prompt_sha256": prompt_meta["sha256"], - "prompt_redacted": not store_full_payloads, - "has_full_prompt": store_full_payloads and bool(prompt), + "prompt_redacted": True, + "has_full_prompt": False, "response_preview": response_meta["preview"], "response_size": response_meta["size"], "response_sha256": response_meta["sha256"], - "response_redacted": not store_full_payloads, - "has_full_response": store_full_payloads and bool(response_content), + "response_redacted": True, + "has_full_response": False, "tokens_used": tokens_used, "duration_ms": duration_ms, "success": success, "error": redact_log_text(error, 1000) } if store_full_payloads: - log_entry["prompt_full"] = prompt - log_entry["response_full"] = response_content + self._volatile_payloads[entry_id] = { + "prompt_full": prompt, + "response_full": response_content, + "workflow": workflow, + } + while len(self._volatile_payloads) > self.MAX_LOG_ENTRIES: + self._volatile_payloads.popitem(last=False) # Append to log file + ensure_private_log_file(self._get_log_path()) with open(self._get_log_path(), "a", encoding="utf-8") as f: f.write(json.dumps(log_entry) + "\n") @@ -211,8 +232,7 @@ async def _trim_log_if_needed(self) -> None: if len(lines) > self.MAX_LOG_ENTRIES: # Keep only the most recent entries lines = lines[-self.MAX_LOG_ENTRIES:] - with open(self._get_log_path(), "w", encoding="utf-8") as f: - f.writelines(lines) + atomic_write_log_lines(self._get_log_path(), lines) logger.debug(f"Trimmed autonomous API log to {self.MAX_LOG_ENTRIES} entries") except Exception as e: @@ -230,6 +250,7 @@ async def get_logs(self, limit: int = 100, include_full: bool = True) -> List[Di """ async with self._lock: try: + self._prepare_active_root() log_path = self._get_log_path() if not os.path.exists(log_path): return [] @@ -243,17 +264,24 @@ async def get_logs(self, limit: int = 100, include_full: bool = True) -> List[Di if line: try: log_entry = json.loads(line) - if not include_full or not system_config.api_log_store_full_payloads: - prompt_full = str(log_entry.pop("prompt_full", "") or "") - response_full = str(log_entry.pop("response_full", "") or "") - log_entry["prompt_size"] = int(log_entry.get("prompt_size") or len(prompt_full)) - log_entry["response_size"] = int(log_entry.get("response_size") or len(response_full)) + prompt_full = str(log_entry.pop("prompt_full", "") or "") + response_full = str(log_entry.pop("response_full", "") or "") + log_entry["prompt_size"] = int(log_entry.get("prompt_size") or len(prompt_full)) + log_entry["response_size"] = int(log_entry.get("response_size") or len(response_full)) + cached = self._volatile_payloads.get(str(log_entry.get("entry_id") or "")) + if ( + include_full + and cached + and system_config.api_log_store_full_payloads + and not system_config.generic_mode + ): + log_entry["prompt_full"] = cached["prompt_full"] + log_entry["response_full"] = cached["response_full"] + log_entry["has_full_prompt"] = bool(cached["prompt_full"]) + log_entry["has_full_response"] = bool(cached["response_full"]) + else: log_entry["has_full_prompt"] = False log_entry["has_full_response"] = False - if prompt_full and not log_entry.get("prompt_sha256"): - log_entry["prompt_sha256"] = hashlib.sha256(prompt_full.encode("utf-8", errors="replace")).hexdigest() - if response_full and not log_entry.get("response_sha256"): - log_entry["response_sha256"] = hashlib.sha256(response_full.encode("utf-8", errors="replace")).hexdigest() logs.append(log_entry) except json.JSONDecodeError: continue @@ -282,6 +310,7 @@ async def clear_logs(self, workflow: Optional[str] = None) -> None: """Clear autonomous API logs, optionally scoped to one workflow.""" async with self._lock: try: + self._prepare_active_root() if workflow: log_path = self._get_log_path() if not os.path.exists(log_path): @@ -303,13 +332,17 @@ async def clear_logs(self, workflow: Optional[str] = None) -> None: if self._entry_workflow(entry) != workflow: retained_lines.append(line) - with open(log_path, "w", encoding="utf-8") as f: - f.writelines(retained_lines) + self._volatile_payloads = OrderedDict( + (entry_id, payload) + for entry_id, payload in self._volatile_payloads.items() + if payload.get("workflow") != workflow + ) + atomic_write_log_lines(log_path, retained_lines) logger.info("Autonomous API logs cleared for workflow %s", workflow) return - with open(self._get_log_path(), "w", encoding="utf-8") as f: - f.write("") + self._volatile_payloads.clear() + atomic_write_log_lines(self._get_log_path(), []) logger.info("Autonomous API logs cleared") except Exception as e: logger.error(f"Failed to clear autonomous API logs: {e}") diff --git a/backend/autonomous/memory/brainstorm_memory.py b/backend/autonomous/memory/brainstorm_memory.py index 0ba7f38..946ebd1 100644 --- a/backend/autonomous/memory/brainstorm_memory.py +++ b/backend/autonomous/memory/brainstorm_memory.py @@ -15,6 +15,7 @@ from backend.shared.log_redaction import redact_log_text from backend.shared.models import BrainstormMetadata from backend.shared.path_safety import resolve_path_within_root, validate_single_path_component +from backend.autonomous.memory.proof_database import is_duplicate_novel_tier logger = logging.getLogger(__name__) @@ -124,6 +125,7 @@ async def create_brainstorm(self, topic_id: str, topic_prompt: str) -> Brainstor topic_prompt=topic_prompt, status="in_progress", submission_count=0, + total_acceptances=0, created_at=datetime.now(), last_activity=datetime.now(), papers_generated=[] @@ -274,6 +276,7 @@ async def add_submission(self, topic_id: str, content: str, submission_number: i metadata = await self.get_metadata(topic_id) if metadata: metadata.submission_count += 1 + metadata.total_acceptances += 1 metadata.last_activity = datetime.now() await self._save_metadata(metadata) @@ -349,10 +352,14 @@ async def append_proofs_section(self, topic_id: str, proofs_data: Any) -> bool: theorem_statement = str(getattr(proof, "theorem_statement", "") or proof.get("theorem_statement", "")).strip() proof_id = str(getattr(proof, "proof_id", "") or proof.get("proof_id", "")).strip() novel = bool(getattr(proof, "novel", False) if hasattr(proof, "novel") else proof.get("novel", False)) + novelty_tier = str(getattr(proof, "novelty_tier", "") or proof.get("novelty_tier", "")).strip() lean_code = str(getattr(proof, "lean_code", "") or proof.get("lean_code", "")).strip() if proof_id and proof_id in existing_ids: continue - status = "Verified (Novel)" if novel else "Verified (Known)" + if is_duplicate_novel_tier(novelty_tier): + status = "Verified (Duplicate Novel)" + else: + status = "Verified (Novel)" if novel else "Verified (Known)" lines.extend( [ @@ -640,68 +647,6 @@ async def clear_submitter_rejections(self, topic_id: str, submitter_id: int) -> # Note: Completion feedback methods moved to autonomous_rejection_logs.py # to avoid duplication and maintain single source of truth - # ======================================================================== - # TOPIC COMBINATION - # ======================================================================== - - async def combine_topics( - self, - new_topic_id: str, - new_topic_prompt: str, - source_topic_ids: List[str] - ) -> Optional[BrainstormMetadata]: - """ - Combine multiple brainstorm topics into a new one. - Merges all submissions from source topics. - """ - async with self._lock: - # Create new brainstorm - metadata = BrainstormMetadata( - topic_id=new_topic_id, - topic_prompt=new_topic_prompt, - status="in_progress", - submission_count=0, - created_at=datetime.now(), - last_activity=datetime.now(), - papers_generated=[] - ) - - # Collect all papers from source topics - for source_id in source_topic_ids: - source_meta = await self.get_metadata(source_id) - if source_meta: - metadata.papers_generated.extend(source_meta.papers_generated) - - # Remove duplicates - metadata.papers_generated = list(set(metadata.papers_generated)) - - # Create empty database file - # NOTE: Do NOT write header comments here - they get interpreted as submission content - # by the fallback parsing in shared_training.py - db_path = self._get_database_path(new_topic_id) - async with aiofiles.open(db_path, 'w', encoding='utf-8') as f: - await f.write("") # Empty file - submissions will be added below - - # Merge submissions from all source topics - submission_counter = 0 - for source_id in source_topic_ids: - submissions = await self.get_submissions_list(source_id) - for sub in submissions: - submission_counter += 1 - async with aiofiles.open(db_path, 'a', encoding='utf-8') as f: - await f.write(f"\n{'=' * 80}\n") - await f.write(f"SUBMISSION #{submission_counter} | Accepted: {datetime.now().isoformat()}\n") - await f.write(f"(Originally from {source_id})\n") - await f.write(f"{'=' * 80}\n\n") - await f.write(sub['content']) - await f.write("\n") - - metadata.submission_count = submission_counter - await self._save_metadata(metadata) - - logger.info(f"Combined {len(source_topic_ids)} topics into {new_topic_id} with {submission_counter} submissions") - return metadata - # ======================================================================== # DELETE OPERATIONS # ======================================================================== diff --git a/backend/autonomous/memory/paper_library.py b/backend/autonomous/memory/paper_library.py index 12d069e..f5ca747 100644 --- a/backend/autonomous/memory/paper_library.py +++ b/backend/autonomous/memory/paper_library.py @@ -292,6 +292,7 @@ def _format_verified_proof_entry(cls, proof: Any, source_context: str = "") -> s "mathematical_discovery": "Mathematical Discovery", "novel_variant": "Novel Reformulation", "novel_formulation": "Novel Formalization", + "duplicate_novel": "Duplicate Novel", } novelty_label = tier_labels.get(novelty_tier, "Novel" if novel else "Known") context_suffix = f"; carried in from {source_context}" if source_context else "" diff --git a/backend/autonomous/memory/proof_database.py b/backend/autonomous/memory/proof_database.py index bdf303e..af13ee7 100644 --- a/backend/autonomous/memory/proof_database.py +++ b/backend/autonomous/memory/proof_database.py @@ -9,6 +9,7 @@ import logging import re import shutil +import uuid from datetime import datetime from pathlib import Path from typing import Dict, Any, List, Optional @@ -18,11 +19,64 @@ from backend.shared.config import system_config from backend.shared.log_redaction import redact_log_text from backend.shared.models import FailedProofCandidate, ProofCandidate, ProofRecord -from backend.shared.path_safety import resolve_path_within_root, validate_single_path_component +from backend.shared.path_safety import resolve_filename_within_root, validate_single_path_component +from backend.shared.proof_identity import canonical_proof_identity from backend.autonomous.prompts.proof_prompts import format_failure_hints_for_injection logger = logging.getLogger(__name__) +DUPLICATE_NOVEL_TIER = "duplicate_novel" +NOT_NOVEL_TIER = "not_novel" +PROOF_LIBRARY_CATEGORIES = frozenset({"novel", "duplicate_novel", "not_novel", "all"}) +PROMPT_INJECTION_NOVEL_TIERS = frozenset( + { + "novel_formulation", + "novel_variant", + "mathematical_discovery", + "major_mathematical_discovery", + } +) + + +def is_duplicate_novel_tier(novelty_tier: str) -> bool: + return str(novelty_tier or "").strip().lower() == DUPLICATE_NOVEL_TIER + + +def is_not_novel_tier(novelty_tier: str) -> bool: + return str(novelty_tier or "").strip().lower() == NOT_NOVEL_TIER + + +def is_syntheticlib_novel_tier(novelty_tier: str) -> bool: + return not is_not_novel_tier(novelty_tier) + + +def is_prompt_injection_novel_tier(novelty_tier: str) -> bool: + return str(novelty_tier or "").strip().lower() in PROMPT_INJECTION_NOVEL_TIERS + + +def normalize_proof_library_category(category: Optional[str] = None, novel_only: Optional[bool] = None) -> str: + normalized = str(category or "").strip().lower() + if normalized in PROOF_LIBRARY_CATEGORIES: + return normalized + if novel_only is None: + return "novel" + return "novel" if novel_only else "all" + + +def proof_matches_library_category(proof_data: Dict[str, Any], category: str) -> bool: + normalized_category = normalize_proof_library_category(category, None) + if normalized_category == "all": + return True + novelty_tier = str(proof_data.get("novelty_tier") or "").strip().lower() + if normalized_category == "duplicate_novel": + return novelty_tier == DUPLICATE_NOVEL_TIER + if normalized_category == "not_novel": + return novelty_tier == NOT_NOVEL_TIER or (not novelty_tier and not bool(proof_data.get("novel"))) + return ( + bool(proof_data.get("novel")) + and (is_prompt_injection_novel_tier(novelty_tier) or not novelty_tier) + ) + class ProofDatabase: """ @@ -37,6 +91,8 @@ class ProofDatabase: def __init__(self) -> None: self._lock = asyncio.Lock() self._base_dir = Path(system_config.data_dir) / "proofs" + self._root_relative_default: Optional[str] = "proofs" + self._root_generation = system_config.runtime_root_generation self._session_manager = None self._index_data: Optional[Dict[str, Any]] = None self._mathlib_reverse_index: Dict[str, List[str]] = {} @@ -47,8 +103,11 @@ def set_session_manager(self, session_manager) -> None: self._session_manager = session_manager if session_manager and session_manager.is_session_active: self._base_dir = session_manager.get_proofs_dir() + self._root_relative_default = None else: self._base_dir = Path(system_config.data_dir) / "proofs" + self._root_relative_default = "proofs" + self._root_generation = system_config.runtime_root_generation self._index_data = None logger.info("Proof database using path: %s", self._base_dir) @@ -56,27 +115,64 @@ def set_base_dir(self, base_dir: Path) -> None: """Use a fixed proof-storage directory independent of autonomous sessions.""" self._session_manager = None self._base_dir = Path(base_dir) + data_root = Path(system_config.data_dir).resolve(strict=False) + resolved = self._base_dir.resolve(strict=False) + try: + relative = resolved.relative_to(data_root) + except ValueError: + self._root_relative_default = None + else: + self._root_relative_default = str(relative) + self._root_generation = system_config.runtime_root_generation self._index_data = None logger.info("Proof database using fixed path: %s", self._base_dir) def _safe_proof_id(self, proof_id: str) -> str: return validate_single_path_component(proof_id, "proof ID") + def _resolve_storage_path(self, filename: str) -> Path: + """Resolve one generated filename beneath the active proof-store root.""" + self._refresh_runtime_root() + return resolve_filename_within_root( + self._base_dir, + filename, + "proof storage filename", + ) + + def _refresh_runtime_root(self) -> None: + if ( + self._session_manager is None + and self._root_relative_default is not None + and self._root_generation != system_config.runtime_root_generation + ): + self._base_dir = Path(system_config.data_dir) / self._root_relative_default + self._root_generation = system_config.runtime_root_generation + self._index_data = None + def _get_index_path(self) -> Path: + self._refresh_runtime_root() return self._base_dir / "proofs_index.json" def _get_record_path(self, proof_id: str) -> Path: - return self._base_dir / f"proof_{self._safe_proof_id(proof_id)}.json" + safe_id = self._safe_proof_id(proof_id) + return self._resolve_storage_path(f"proof_{safe_id}.json") def _get_lean_path(self, proof_id: str) -> Path: - return self._base_dir / f"proof_{self._safe_proof_id(proof_id)}_lean.lean" + safe_id = self._safe_proof_id(proof_id) + return self._resolve_storage_path(f"proof_{safe_id}_lean.lean") def _get_failed_dir(self) -> Path: + self._refresh_runtime_root() return self._base_dir / "failed" def _get_failed_candidates_path(self, source_brainstorm_id: str) -> Path: safe_id = validate_single_path_component(source_brainstorm_id, "brainstorm ID") - return self._get_failed_dir() / f"{safe_id}.json" + failed_dir = self._get_failed_dir() + return resolve_filename_within_root( + failed_dir, + f"{safe_id}.json", + "failed candidate filename", + ) def _default_index(self) -> Dict[str, Any]: return { @@ -84,6 +180,18 @@ def _default_index(self) -> Dict[str, Any]: "proofs": [], } + async def get_or_create_active_run_id(self) -> str: + """Return the durable explicit run ID owned by this proof database.""" + async with self._lock: + if self._index_data is None: + await self._load_index() + run_id = str(self._index_data.get("active_run_id") or "").strip() + if not run_id: + run_id = f"manual-{uuid.uuid4().hex}" + self._index_data["active_run_id"] = run_id + await self._save_index() + return run_id + def _rebuild_reverse_indexes(self) -> None: self._mathlib_reverse_index = {} self._mathlib_reverse_short_index = {} @@ -110,6 +218,7 @@ def _rebuild_reverse_indexes(self) -> None: self._mathlib_reverse_short_index[short_name].append(proof_id) def _rebuild_index_from_record_files_sync(self) -> Dict[str, Any]: + self._refresh_runtime_root() proofs: List[Dict[str, Any]] = [] for record_path in self._base_dir.glob("proof_*.json"): if record_path.name.endswith("_metadata.json"): @@ -138,6 +247,8 @@ async def initialize(self) -> None: """Ensure storage exists and load the index.""" if self._session_manager and self._session_manager.is_session_active: self._base_dir = self._session_manager.get_proofs_dir() + else: + self._refresh_runtime_root() self._base_dir.mkdir(parents=True, exist_ok=True) self._get_failed_dir().mkdir(parents=True, exist_ok=True) @@ -241,36 +352,79 @@ async def _save_failed_candidates( async with aiofiles.open(failed_path, "w", encoding="utf-8") as handle: await handle.write(json.dumps(payload, indent=2)) + async def clear_failed_candidates(self) -> None: + """Remove active failed proof retry hints without touching verified proofs.""" + async with self._lock: + failed_dir = self._get_failed_dir() + if failed_dir.exists(): + await asyncio.to_thread(shutil.rmtree, failed_dir, True) + failed_dir.mkdir(parents=True, exist_ok=True) + async def add_proof(self, record: ProofRecord) -> ProofRecord: """Persist a proof record and return the stored copy.""" stored_record, _duplicate = await self.add_proof_if_absent(record) return stored_record + async def _persist_record_files( + self, + stored_record: ProofRecord, + serialized: Dict[str, Any], + ) -> None: + """Persist one proof's metadata and Lean source through validated paths.""" + record_path = self._get_record_path(stored_record.proof_id) + lean_path = self._get_lean_path(stored_record.proof_id) + async with aiofiles.open(record_path, "w", encoding="utf-8") as handle: + await handle.write(json.dumps(serialized, indent=2)) + async with aiofiles.open(lean_path, "w", encoding="utf-8") as handle: + await handle.write(stored_record.lean_code) + + async def add_proof_occurrence(self, record: ProofRecord) -> ProofRecord: + """Persist a full record for each newly verified current-run occurrence.""" + async with self._lock: + if self._index_data is None: + await self._load_index() + + proof_id = record.proof_id or f"proof_{self._index_data['next_proof_id']:03d}" + stored_record = record.model_copy(update={"proof_id": proof_id}) + serialized = self._serialize_record(stored_record) + await self._persist_record_files(stored_record, serialized) + + proofs = [ + proof + for proof in self._index_data.get("proofs", []) + if proof.get("proof_id") != proof_id + ] + proofs.append(serialized) + proofs.sort(key=lambda proof: proof.get("created_at", ""), reverse=True) + self._index_data["proofs"] = proofs + current_number = self._index_data.get("next_proof_id", 1) + self._index_data["next_proof_id"] = max(current_number + 1, len(proofs) + 1) + self._rebuild_reverse_indexes() + await self._save_index() + return stored_record + async def add_proof_if_absent(self, record: ProofRecord) -> tuple[ProofRecord, bool]: """Persist a proof record unless an identical source/theorem/code exists.""" async with self._lock: if self._index_data is None: await self._load_index() - normalized_statement = " ".join((record.theorem_statement or "").split()) - normalized_code = "\n".join((record.lean_code or "").strip().splitlines()) + identity = canonical_proof_identity(record.theorem_statement, record.lean_code) for existing in self._index_data.get("proofs", []): if existing.get("source_type") != record.source_type or existing.get("source_id") != record.source_id: continue - if " ".join(str(existing.get("theorem_statement") or "").split()) != normalized_statement: - continue - if "\n".join(str(existing.get("lean_code") or "").strip().splitlines()) != normalized_code: + existing_identity = canonical_proof_identity( + str(existing.get("theorem_statement") or ""), + str(existing.get("lean_code") or ""), + ) + if existing_identity.key != identity.key: continue return self._deserialize_record(existing), True proof_id = record.proof_id or f"proof_{self._index_data['next_proof_id']:03d}" stored_record = record.model_copy(update={"proof_id": proof_id}) serialized = self._serialize_record(stored_record) - - async with aiofiles.open(self._get_record_path(proof_id), "w", encoding="utf-8") as handle: - await handle.write(json.dumps(serialized, indent=2)) - async with aiofiles.open(self._get_lean_path(proof_id), "w", encoding="utf-8") as handle: - await handle.write(stored_record.lean_code) + await self._persist_record_files(stored_record, serialized) proofs = [ proof @@ -459,7 +613,21 @@ async def get_all_proofs(self, novel_only: Optional[bool] = None) -> List[ProofR ] if novel_only is None: return proofs - return [proof for proof in proofs if proof.novel is novel_only] + if novel_only: + return [ + proof for proof in proofs + if proof.novel and ( + is_prompt_injection_novel_tier(proof.novelty_tier) + or not str(proof.novelty_tier or "").strip() + ) + ] + return [ + proof for proof in proofs + if not proof.novel or ( + bool(str(proof.novelty_tier or "").strip()) + and not is_prompt_injection_novel_tier(proof.novelty_tier) + ) + ] async def update_proof_dependencies(self, proof_id: str, dependencies) -> Optional[ProofRecord]: """Persist a new dependency list for an existing proof record.""" @@ -623,11 +791,23 @@ def count_proofs(self) -> Dict[str, int]: """Return proof counts for display and prompt gating.""" self._ensure_index_loaded_sync() proofs = self._index_data.get("proofs", []) if self._index_data else [] - novel_count = sum(1 for proof in proofs if proof.get("novel")) + duplicate_novel_count = sum( + 1 for proof in proofs if is_duplicate_novel_tier(proof.get("novelty_tier", "")) + ) + prompt_novel_count = sum( + 1 for proof in proofs if proof.get("novel") and not is_duplicate_novel_tier(proof.get("novelty_tier", "")) + ) + syntheticlib_novel_count = prompt_novel_count + duplicate_novel_count + not_novel_count = sum( + 1 for proof in proofs if is_not_novel_tier(proof.get("novelty_tier", NOT_NOVEL_TIER)) + ) return { "total": len(proofs), - "novel": novel_count, - "known": len(proofs) - novel_count, + "novel": prompt_novel_count, + "syntheticlib_novel": syntheticlib_novel_count, + "duplicate_novel": duplicate_novel_count, + "not_novel": not_novel_count, + "known": len(proofs) - syntheticlib_novel_count, } def get_known_proofs_summary_for_browsing( @@ -654,7 +834,10 @@ def get_known_proofs_summary_for_browsing( """ self._ensure_index_loaded_sync() proofs = self._index_data.get("proofs", []) if self._index_data else [] - known_proofs = [p for p in proofs if not p.get("novel")] + known_proofs = [ + p for p in proofs + if not p.get("novel") or is_duplicate_novel_tier(p.get("novelty_tier", "")) + ] if source_id: known_proofs = [p for p in known_proofs if p.get("source_id") == source_id] @@ -688,7 +871,13 @@ def get_novel_proofs_for_injection(self) -> str: """Format the novel proofs block for highest-priority prompt injection.""" self._ensure_index_loaded_sync() proofs = self._index_data.get("proofs", []) if self._index_data else [] - novel_proofs = [proof for proof in proofs if proof.get("novel")] + novel_proofs = [ + proof for proof in proofs + if proof.get("novel") and ( + is_prompt_injection_novel_tier(proof.get("novelty_tier", "")) + or not str(proof.get("novelty_tier") or "").strip() + ) + ] if not novel_proofs: return "" @@ -751,17 +940,22 @@ async def inject_failure_hints_into_prompt( return hints_block return f"{hints_block}\n\n{prompt}" - async def list_proof_library(self, novel_only: bool = True) -> List[Dict[str, Any]]: + async def list_proof_library( + self, + novel_only: Optional[bool] = True, + category: Optional[str] = None, + ) -> List[Dict[str, Any]]: """List all proofs across all sessions (legacy + session-based) for the proof library. Mirrors the cross-session listing pattern used by PaperLibrary.list_history_papers(). """ + normalized_category = normalize_proof_library_category(category, novel_only) all_proofs: List[Dict[str, Any]] = [] legacy_proofs_dir = Path(system_config.data_dir) / "proofs" if legacy_proofs_dir.exists(): all_proofs.extend( - await self._list_proofs_from_directory(legacy_proofs_dir, "legacy", novel_only) + await self._list_proofs_from_directory(legacy_proofs_dir, "legacy", normalized_category) ) sessions_dir = Path(system_config.auto_sessions_base_dir) @@ -773,7 +967,7 @@ async def list_proof_library(self, novel_only: bool = True) -> List[Dict[str, An if not proofs_dir.exists(): continue all_proofs.extend( - await self._list_proofs_from_directory(proofs_dir, session_dir.name, novel_only) + await self._list_proofs_from_directory(proofs_dir, session_dir.name, normalized_category) ) all_proofs.sort(key=lambda p: p.get("created_at") or "", reverse=True) @@ -796,7 +990,9 @@ async def archive_current_run( self._ensure_index_loaded_sync() proof_count = len(self._index_data.get("proofs", []) if self._index_data else []) has_files = self._base_dir.exists() and any(self._base_dir.iterdir()) - if not has_files or proof_count == 0: + failed_dir = self._get_failed_dir() + has_failed_state = failed_dir.exists() and any(failed_dir.iterdir()) + if not has_files or (proof_count == 0 and not has_failed_state): if self._base_dir.exists(): await asyncio.to_thread(shutil.rmtree, self._base_dir, True) self._index_data = self._default_index() @@ -810,7 +1006,8 @@ async def archive_current_run( history_root = Path(history_root) history_root.mkdir(parents=True, exist_ok=True) - base_run_id = f"manual_proofs_{timestamp}" + active_run_id = str(self._index_data.get("active_run_id") or "").strip() + base_run_id = active_run_id or f"manual-{uuid.uuid4().hex}" run_id = base_run_id suffix = 2 while (history_root / run_id).exists(): @@ -823,6 +1020,8 @@ async def archive_current_run( def _copy_active_run() -> None: run_dir.mkdir(parents=True, exist_ok=True) shutil.copytree(self._base_dir, target_proofs_dir) + if proof_count: + shutil.rmtree(target_proofs_dir / "failed", ignore_errors=True) await asyncio.to_thread(_copy_active_run) @@ -835,6 +1034,7 @@ def _copy_active_run() -> None: "created_at": timestamp, "archived_at": datetime.utcnow().isoformat(), "proof_count": proof_count, + "has_failed_state": has_failed_state, } metadata_path = run_dir / "session_metadata.json" await asyncio.to_thread( @@ -854,9 +1054,11 @@ def _copy_active_run() -> None: async def list_proof_library_from_history( self, history_root: Path, - novel_only: bool = True, + novel_only: Optional[bool] = True, + category: Optional[str] = None, ) -> List[Dict[str, Any]]: """List archived manual proof runs without including the active DB.""" + normalized_category = normalize_proof_library_category(category, novel_only) history_root = Path(history_root) all_proofs: List[Dict[str, Any]] = [] if history_root.exists(): @@ -865,13 +1067,13 @@ async def list_proof_library_from_history( if not proofs_dir.exists(): continue all_proofs.extend( - await self._list_proofs_from_directory(proofs_dir, run_dir.name, novel_only) + await self._list_proofs_from_directory(proofs_dir, run_dir.name, normalized_category) ) all_proofs.sort(key=lambda p: p.get("created_at") or "", reverse=True) return all_proofs async def _list_proofs_from_directory( - self, proofs_dir: Path, session_id: str, novel_only: bool + self, proofs_dir: Path, session_id: str, category: str ) -> List[Dict[str, Any]]: """Read the proofs index from a specific directory and return library entries.""" index_path = proofs_dir / "proofs_index.json" @@ -886,20 +1088,28 @@ async def _list_proofs_from_directory( return [] session_metadata_path = proofs_dir.parent / "session_metadata.json" - user_prompt = "" + session_user_prompt = "" + session_run_id = session_id if session_metadata_path.exists(): try: async with aiofiles.open(session_metadata_path, "r", encoding="utf-8") as handle: meta = json.loads(await handle.read()) - user_prompt = meta.get("user_prompt", "") + session_user_prompt = str(meta.get("user_prompt", "") or "").strip() + session_run_id = str( + meta.get("run_id") or meta.get("session_id") or session_id + ).strip() except Exception as exc: logger.debug("Failed to read proof library session metadata at %s: %s", session_metadata_path, exc) results: List[Dict[str, Any]] = [] for proof_data in index_data.get("proofs", []): is_novel = proof_data.get("novel", False) - if novel_only and not is_novel: + if not proof_matches_library_category(proof_data, category): continue + run_id = str(proof_data.get("run_id") or session_run_id or session_id).strip() + user_prompt = str( + proof_data.get("user_prompt") or session_user_prompt or proof_data.get("source_title") or "" + ).strip() results.append({ "library_id": f"{session_id}:{proof_data.get('proof_id', '')}", @@ -911,10 +1121,14 @@ async def _list_proofs_from_directory( "source_type": proof_data.get("source_type", ""), "source_id": proof_data.get("source_id", ""), "source_title": proof_data.get("source_title", ""), + "run_id": run_id, "solver": proof_data.get("solver", "Lean 4"), "novel": is_novel, "novelty_tier": proof_data.get("novelty_tier", "not_novel"), "novelty_reasoning": proof_data.get("novelty_reasoning", ""), + "artifact_purpose": proof_data.get( + "artifact_purpose", "verified_occurrence" + ), "verification_notes": proof_data.get("verification_notes", ""), "attempt_count": proof_data.get("attempt_count", 0), "created_at": proof_data.get("created_at", ""), @@ -989,10 +1203,31 @@ async def get_library_proof_from_directory( else: lean_code = str(proof_data.get("lean_code", "") or "") + session_user_prompt = "" + session_run_id = session_id + metadata_path = proofs_dir.parent / "session_metadata.json" + if metadata_path.exists(): + try: + async with aiofiles.open(str(metadata_path), "r", encoding="utf-8") as handle: + metadata = json.loads(await handle.read()) + session_user_prompt = str(metadata.get("user_prompt", "") or "").strip() + session_run_id = str( + metadata.get("run_id") or metadata.get("session_id") or session_id + ).strip() + except Exception as exc: + logger.debug("Failed to read proof detail session metadata at %s: %s", metadata_path, exc) + return { "library_id": f"{session_id}:{proof_id}", "session_id": session_id, **proof_data, + "run_id": str(proof_data.get("run_id") or session_run_id or session_id).strip(), + "user_prompt": str( + proof_data.get("user_prompt") + or session_user_prompt + or proof_data.get("source_title") + or "" + ).strip(), "lean_code": lean_code, } diff --git a/backend/autonomous/memory/research_metadata.py b/backend/autonomous/memory/research_metadata.py index 010ca8d..316f11f 100644 --- a/backend/autonomous/memory/research_metadata.py +++ b/backend/autonomous/memory/research_metadata.py @@ -490,9 +490,9 @@ async def generate_paper_id(self) -> str: # ======================================================================== async def get_user_prompt(self) -> str: - """Get the user's research prompt.""" + """Get the canonical user-authored research prompt.""" await self._ensure_initialized() - return self._data.get("user_research_prompt", "") + return self._data.get("base_user_research_prompt") or self._data.get("user_research_prompt", "") async def get_base_user_prompt(self) -> str: """Get the original user research prompt before proof framing.""" @@ -500,9 +500,10 @@ async def get_base_user_prompt(self) -> str: return self._data.get("base_user_research_prompt") or self._data.get("user_research_prompt", "") async def set_user_prompt(self, prompt: str) -> None: - """Set the user's research prompt.""" + """Persist only the canonical user-authored research prompt.""" async with self._lock: self._data["user_research_prompt"] = prompt + self._data["base_user_research_prompt"] = prompt await self._save_metadata() async def set_proof_framing_state( @@ -514,10 +515,10 @@ async def set_proof_framing_state( context: str, reasoning: str, ) -> None: - """Persist the proof-framing decision in metadata.""" + """Persist proof framing separately from the canonical user prompt.""" async with self._lock: self._data["base_user_research_prompt"] = base_user_prompt - self._data["user_research_prompt"] = effective_user_prompt + self._data["user_research_prompt"] = base_user_prompt self._data["proof_framing_active"] = active self._data["proof_framing_context"] = context self._data["proof_framing_reasoning"] = reasoning diff --git a/backend/autonomous/prompts/completion_prompts.py b/backend/autonomous/prompts/completion_prompts.py index 01564b3..fcb745b 100644 --- a/backend/autonomous/prompts/completion_prompts.py +++ b/backend/autonomous/prompts/completion_prompts.py @@ -25,7 +25,7 @@ def get_completion_review_system_prompt() -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been established as correct. Assess it with domain- and claim-appropriate rigor. Mathematical reasoning, theorem discovery, proof, and formalization remain first-class whenever relevant. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -35,7 +35,12 @@ def get_completion_review_system_prompt() -> str: Assess whether you have sufficiently explored this brainstorm topic using all available resources (your base knowledge, web search if available, and the brainstorm database), and decide whether to continue or write a paper. CRITICAL UNDERSTANDING: -This is an assessment of topic exploration completeness using all resources at your disposal. Consider whether you can contribute more valuable mathematical insights using your knowledge, web search capabilities (if available), and analysis of what's been covered. +This is an assessment of topic exploration completeness using all resources at your disposal. Consider whether you can contribute more high-impact routes, constraints, mechanisms, proofs, evidence needs, experiments, implementation obstacles, safety issues, or failure modes using your knowledge, web search capabilities (if available), and analysis of what has been covered. + +SYNTHESIS READINESS IS NOT EMPIRICAL DEMONSTRATION: +- For empirical or engineering work, WRITE_PAPER means the brainstorm is ready to synthesize the strongest currently justified proposal, analysis, or test plan +- It does NOT mean a proposed mechanism has been built, an experiment has been run, or a hypothesis has been empirically demonstrated +- Preserve proposed-work and uncertainty language; never upgrade internal ideas into fabricated measurements, artifacts, citations, or completed validation DIRECT-SOLUTION PREFERENCE: - Prefer moving to paper writing once the brainstorm can support the strongest rigorous direct answer currently justified @@ -45,13 +50,13 @@ def get_completion_review_system_prompt() -> str: DECISION CRITERIA: Choose CONTINUE_BRAINSTORM if: -- You can identify specific mathematical areas not yet covered in the submissions that are likely to improve the direct answer -- You have additional rigorous work relevant to the topic (from your knowledge or discoverable via web search) +- You can identify specific major routes, constraints, mechanisms, proof directions, evidence needs, experiments, implementation obstacles, safety issues, or failure modes not yet covered that are likely to improve the direct answer +- You have additional rigorous work relevant to the topic (from your knowledge or discoverable via web search), judged by the standards appropriate to its claim type - The brainstorm would benefit from deeper exploration in specific directions that materially strengthen direct resolution - You can still contribute valuable direct-progress insights using available resources (base knowledge, web search if available) Choose WRITE_PAPER if: -- All major mathematical avenues for this topic have been explored +- All major high-impact solution and research avenues for this topic have been explored sufficiently for synthesis - Additional submissions would likely be redundant with existing content - The brainstorm database is comprehensive enough for a quality paper that gives the strongest currently justified direct answer - Available resources (base knowledge, web search if available) have been sufficiently utilized for this topic @@ -61,7 +66,8 @@ def get_completion_review_system_prompt() -> str: - Be honest about whether you truly have more to contribute - Don't artificially extend brainstorming if exhausted - Don't prematurely end if valuable knowledge remains -- Consider the mathematical depth achieved, not just submission count +- Consider the depth, correctness, provenance, feasibility, and verification potential achieved, not just submission count +- For mathematical work, retain the full standard of sound derivation, proof, or explicit assumptions - Prefer best-answer readiness over breadth for breadth's sake CRITICAL JSON ESCAPE RULES: @@ -79,7 +85,7 @@ def get_completion_review_json_schema() -> str: { "decision": "continue_brainstorm | write_paper", "reasoning": "string - Detailed explanation of assessment", - "suggested_additions": "string - If continue_brainstorm, what mathematical areas remain unexplored (optional)" + "suggested_additions": "string - If continue_brainstorm, what major solution routes or verification needs remain unexplored (optional)" } FIELD REQUIREMENTS: @@ -92,14 +98,14 @@ def get_completion_review_json_schema() -> str: Continue Brainstorm: { "decision": "continue_brainstorm", - "reasoning": "While the brainstorm has covered fundamental aspects of modular forms and their Galois representations, there remain unexplored areas including explicit computational methods, connections to elliptic curves, and applications to specific cases of Langlands correspondence.", - "suggested_additions": "Explore explicit computations of Galois representations attached to modular forms, investigate connections to elliptic curves over number fields, examine specific cases of the Langlands correspondence for GL(2)" + "reasoning": "The brainstorm proposes a low-energy catalyst pathway, but it has not specified a falsifiable degradation mechanism, control conditions, or measurements that would distinguish the hypothesis from competing explanations.", + "suggested_additions": "Define the degradation hypothesis, required controls, measurable outcomes, confounders, and a feasible experiment plan without claiming that any experiment has already been completed." } Write Paper: { "decision": "write_paper", - "reasoning": "The brainstorm has thoroughly explored modular forms, Galois representations, L-functions, automorphic forms, and their interconnections in the context of Langlands program. The database contains 23 high-quality submissions covering theoretical foundations, computational aspects, and specific examples. Further submissions would likely be redundant. A comprehensive paper can now synthesize these insights." + "reasoning": "The brainstorm now contains a concrete resilient-protocol design, explicit network and fault assumptions, safety and liveness arguments, operational constraints, failure modes, and a simulation and implementation verification plan. Further submissions would likely repeat these routes. A paper can synthesize the strongest currently justified proposal, while clearly stating that the prototype and empirical tests remain proposed work rather than completed demonstrations." }""" @@ -121,7 +127,7 @@ def get_completion_self_validation_system_prompt() -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been established as correct. Apply domain- and claim-appropriate rigor; mathematical reasoning and proof remain first-class whenever relevant. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -131,7 +137,7 @@ def get_completion_self_validation_system_prompt() -> str: Review your OWN completion assessment and validate whether it is accurate. CRITICAL UNDERSTANDING: -You just assessed whether your internal knowledge on a brainstorm topic has been sufficiently explored. Now you must validate that assessment. This is a self-check to ensure accuracy. +You just assessed whether your available knowledge and resources on a brainstorm topic have been sufficiently explored for synthesis. Now you must validate that assessment. This is a self-check to ensure accuracy, not a claim that proposed empirical or engineering work has already been demonstrated. VALIDATION CRITERIA: @@ -140,6 +146,7 @@ def get_completion_self_validation_system_prompt() -> str: - If you said "continue_brainstorm": You genuinely have more valuable direct-progress insights to contribute using available resources - If you said "write_paper": You genuinely cannot think of significant new contributions that would materially strengthen the direct answer - The reasoning in your assessment is sound and honest +- For empirical or engineering work, a write_paper decision means synthesis-ready only; it does not assert completed experiments, measurements, implementations, or validation Validate as FALSE if: - Upon reflection, the assessment was CLEARLY incorrect @@ -179,19 +186,19 @@ def get_completion_self_validation_json_schema() -> str: Validated True (assessment was reasonable): { "validated": true, - "reasoning": "The completion assessment accurately reflects the current state of the brainstorm. The database has covered the major mathematical avenues for this topic, and the decision is well-reasoned. While more could theoretically be added, the assessment is sound and should be accepted." + "reasoning": "The completion assessment accurately reflects the current state of the brainstorm. The major answer-bearing routes, constraints, and verification needs are covered well enough for honest synthesis. This confirms readiness to write, not empirical demonstration of the proposed work." } Validated True (continue decision was reasonable): { "validated": true, - "reasoning": "The decision to continue brainstorming is valid. The suggested additions represent genuine unexplored areas that would meaningfully enhance the research. The assessment is accurate." + "reasoning": "The decision to continue brainstorming is valid. The suggested additions identify concrete unresolved mechanism, evidence, and failure-mode questions that would meaningfully improve the solution. The assessment is accurate." } Validated False (ONLY use when clearly incorrect): { "validated": false, - "reasoning": "Upon reflection, I identified a SPECIFIC error: the assessment claimed topic X was covered, but submissions #12 and #15 only touched on it tangentially. The core theoretical framework for X remains unexplored. This is a concrete gap that invalidates my write_paper decision." + "reasoning": "Upon reflection, I identified a SPECIFIC error: the assessment treated the proposed catalyst mechanism as synthesis-ready, but the database has no falsifiable discriminator, control design, or measurement plan. This concrete gap invalidates my write_paper decision without implying that experiments must already have been performed." } NOTE: Default to validated=true unless you identify a SPECIFIC, CONCRETE error in your reasoning. Vague concerns like 'more could be added' are NOT sufficient to invalidate.""" diff --git a/backend/autonomous/prompts/final_answer_prompts.py b/backend/autonomous/prompts/final_answer_prompts.py index 5bafe0a..e319f95 100644 --- a/backend/autonomous/prompts/final_answer_prompts.py +++ b/backend/autonomous/prompts/final_answer_prompts.py @@ -35,14 +35,14 @@ def get_certainty_assessment_system_prompt() -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been proven correct. Apply the rigor and evidence standard appropriate to each domain and claim type. Distinguish proven facts, supported conclusions, proposals, hypotheses, and work that still requires validation. Mathematical reasoning and formal proof remain first-class when relevant. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. --- YOUR TASK: -Review all existing research papers and determine what can be answered WITH CERTAINTY - without speculation or theoretical hand-waving. +Review all existing research papers and determine what answer is defensible at the applicable rigor and evidence standard, without speculation, fabrication, or unsupported certainty. DIRECT-ANSWER-FIRST REQUIREMENT: - Identify the strongest direct answer the papers justify, not just nearby facts @@ -52,33 +52,34 @@ def get_certainty_assessment_system_prompt() -> str: 1. TOTAL_ANSWER - The user's question can be FULLY answered with high confidence - All aspects of the question are addressed by the papers - - The answer is mathematically rigorous and well-supported - - No significant gaps or speculation needed + - The complete answer is well-supported at the standard applicable to its claims + - No significant proof, evidence, implementation, or validation gaps remain 2. PARTIAL_ANSWER - The question can be partially answered with certainty - - Some aspects are well-established + - Identified portions are well-supported - Other aspects remain uncertain or unexplored - A meaningful but incomplete answer is possible 3. NO_ANSWER_KNOWN - The existing research doesn't provide an answer - Papers explore related topics but don't address the core question - - More research is needed before any answer can be given + - Available papers do not support a defensible answer - The system should continue research (Tier 3 will not complete) -4. APPEARS_IMPOSSIBLE - The question appears mathematically impossible - - Research supports that the question as posed has no valid answer - - The question as posed has no valid answer - - Can still provide a paper explaining why it's impossible +4. APPEARS_IMPOSSIBLE - The objective appears impossible or infeasible for a clearly reasoned reason + - It may be mathematically impossible, physically infeasible, internally inconsistent, prohibited by stated constraints, or otherwise unsupported as posed + - The conclusion must be justified at the applicable standard, not inferred merely from exhausted idea space + - A final answer may explain the impossibility, infeasibility, or contradiction 5. OTHER - Special cases that don't fit the above - Explain what makes this case unique CRITICAL REQUIREMENTS: - Base assessment ONLY on the papers you've reviewed -- Identify what is KNOWN WITH CERTAINTY vs what is SPECULATIVE +- Identify proven facts, supported conclusions, proposals, hypotheses, and required validation separately +- Never treat an invention, implementation, or experiment as demonstrated merely because it is proposed or because the explored idea space appears exhausted - Do not claim certainty where uncertainty exists -- Summarize the key certainties that have been established -- State the best direct answer those certainties support +- Summarize the strongest defensible findings and their evidence status +- State the best direct answer those findings support CRITICAL JSON ESCAPE RULES: 1. Backslashes: ALWAYS use double backslash (\\\\) for any backslash in your text @@ -93,23 +94,23 @@ def get_certainty_assessment_json_schema() -> str: return """REQUIRED JSON FORMAT: { "certainty_level": "total_answer | partial_answer | no_answer_known | appears_impossible | other", - "known_certainties_summary": "string - Detailed summary of what is established with certainty from the papers", + "known_certainties_summary": "string - Detailed summary of the strongest defensible findings and their evidence status", "reasoning": "string - Why this certainty level was chosen, referencing specific papers" } FIELD REQUIREMENTS: - certainty_level: MUST be one of the five options -- known_certainties_summary: ALWAYS required - what can we say FOR CERTAIN +- known_certainties_summary: ALWAYS required - distinguish established results, supported conclusions, proposals, hypotheses, and required validation - reasoning: ALWAYS required - justify your assessment -EXAMPLE (Partial Answer): +EXAMPLE (Partial Answer - Engineering): { "certainty_level": "partial_answer", - "known_certainties_summary": "From the research papers, we have established with certainty: (1) the answer to the core circle-squaring question (paper_003), (2) the role of pi's transcendence (paper_007), (3) the connection between constructibility and algebraic field extensions (paper_012). However, the specific computational bounds requested by the user remain unexplored.", - "reasoning": "Papers 003, 007, and 012 provide rigorous support for the core answer. However, the user's question also asks about approximation algorithms, which none of the papers address. Therefore, only a partial answer can be given with certainty." + "known_certainties_summary": "Papers 003 and 007 support the proposed mechanism through analysis and simulation, and paper 012 identifies credible failure modes. No physical prototype or controlled experiment has yet validated the claimed efficiency.", + "reasoning": "The mechanism and constraints are sufficiently supported to present a defensible design proposal, but the requested performance claim remains a hypothesis pending empirical validation. Therefore only a partial answer is justified." } -EXAMPLE (Total Answer): +EXAMPLE (Total Answer - Mathematical): { "certainty_level": "total_answer", "known_certainties_summary": "The user's question about the Lindemann-Weierstrass theorem is fully addressed. Paper_005 provides the complete proof. Paper_008 establishes all necessary preliminary results. Paper_015 addresses the specific applications the user asked about. All components of a comprehensive answer are present.", @@ -132,7 +133,7 @@ def get_certainty_validator_system_prompt() -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been proven correct. Judge each claim under the rigor and evidence standard appropriate to its domain. Mathematical claims require sound derivation or proof; empirical, artifact, engineering, software, and causal claims require corresponding evidence, provenance, feasibility reasoning, and validation. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -146,6 +147,7 @@ def get_certainty_validator_system_prompt() -> str: ACCEPT the assessment if: - The certainty level accurately reflects what the papers establish - The known certainties summary correctly identifies established facts +- Proposals and hypotheses are not mislabeled as demonstrated results - The reasoning properly references the papers - No overclaiming certainty where uncertainty exists - No underclaiming (missing obvious certainties) @@ -157,6 +159,7 @@ def get_certainty_validator_system_prompt() -> str: - Reasoning doesn't properly support the conclusion - Important certainties from papers are missed - Speculation is presented as certainty +- Exhausted idea space is treated as proof that an invention works or an experiment succeeded CRITICAL JSON ESCAPE RULES: 1. Backslashes: ALWAYS use double backslash (\\\\) for any backslash in your text @@ -199,7 +202,7 @@ def get_format_selection_system_prompt() -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been proven correct. Apply domain- and claim-appropriate rigor, preserve the evidence status of every conclusion, and retain mathematical reasoning and formal proof as first-class modalities whenever relevant. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -219,17 +222,18 @@ def get_format_selection_system_prompt() -> str: - Includes existing papers as body chapters - New Introduction and Conclusion papers frame the collection - May include "gap papers" for missing content -- Best for complex questions requiring multiple perspectives -- Appropriate when existing papers cover different aspects comprehensively +- Best when genuinely independent solution components require separate treatment +- Appropriate when those components have a real dependency structure that cannot be presented clearly in one paper DECISION FACTORS: - Complexity of the user's question - Number and diversity of relevant papers - Whether a single coherent narrative is possible -- Whether the papers naturally form a cohesive volume +- Whether independent mechanisms, evidence, implementations, proofs, validations, or risk analyses genuinely require separate chapters - The certainty level from Phase 1 - Prefer short form whenever one paper can honestly provide the strongest direct answer - Choose long form only when multiple chapters are genuinely necessary to deliver that answer well +- The number of source papers alone does not justify a volume CRITICAL JSON ESCAPE RULES: 1. Backslashes: ALWAYS use double backslash (\\\\) for any backslash in your text @@ -251,13 +255,13 @@ def get_format_selection_json_schema() -> str: - answer_format: MUST be "short_form" or "long_form" - reasoning: ALWAYS required -EXAMPLE (Short Form): +EXAMPLE (Short Form - Engineering): { "answer_format": "short_form", - "reasoning": "The user's question about the transcendence of pi can be answered comprehensively in a single paper. Papers 003, 005, and 007 cover complementary aspects that can be synthesized into a coherent narrative. The question has a focused scope that doesn't warrant a multi-chapter volume." + "reasoning": "The proposed low-energy treatment mechanism, feasibility constraints, and validation plan form one tightly coupled solution that can be presented honestly and coherently in a single paper. The number of source papers alone does not justify a volume." } -EXAMPLE (Long Form): +EXAMPLE (Long Form - Mathematical): { "answer_format": "long_form", "reasoning": "The user's question about the Langlands program requires addressing multiple deep topics: automorphic forms, Galois representations, L-functions, and their connections. Papers 002, 005, 008, 011, and 015 each cover distinct essential aspects. A volume with these as chapters, plus an introduction explaining how they connect and a conclusion summarizing the current state of knowledge, will provide the most complete answer." @@ -279,7 +283,7 @@ def get_format_validator_system_prompt() -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been proven correct. Validate the format against the actual dependency structure of the answer and the rigor and evidence standard appropriate to its claims. Mathematics and formal proof remain fully available when relevant. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -294,12 +298,13 @@ def get_format_validator_system_prompt() -> str: - The format appropriately matches the scope of the question - The reasoning is sound - Short form is chosen only when a single paper suffices -- Long form is chosen when multiple perspectives are needed +- Long form is chosen only when genuinely independent solution components need separate chapters - The choice preserves the clearest path to a direct answer REJECT the selection if: - Short form is chosen for a question requiring extensive treatment - Long form is chosen unnecessarily for a focused question +- Long form is chosen merely because many papers exist - The reasoning doesn't support the choice - The selection ignores important factors - The selection adds unnecessary structural breadth instead of optimizing for a direct answer @@ -345,7 +350,7 @@ def get_final_paper_title_system_prompt() -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been proven correct. Organize the strongest defensible direct answer using the rigor and evidence standard appropriate to each component. Mathematical reasoning, theorems, and formal proofs remain first-class when relevant. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -354,14 +359,14 @@ def get_final_paper_title_system_prompt() -> str: YOUR TASK: Choose a title that: 1. DIRECTLY and TRANSPARENTLY answers or addresses the user's question -2. Reflects the known certainties from the research -3. Is appropriate for a mathematical research paper -4. Makes clear this is a definitive answer, not exploratory research +2. Reflects the strongest defensible findings and their evidence status +3. Is appropriate for the solution form and domain +4. Makes the answer's actual level of support clear TITLE GUIDELINES: - The title should make the answer's conclusion clear when possible - The title can indicate the answer when the papers justify one -- Be specific about the mathematical content +- Be specific about the mechanism, evidence, implementation, theorem, proof, limitation, or other central solution content - Avoid vague or overly general titles CRITICAL JSON ESCAPE RULES: @@ -411,7 +416,7 @@ def get_volume_organization_system_prompt() -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been proven correct. Validate the dependency structure and evidence status of the proposed answer under domain- and claim-appropriate rigor. Mathematical results and formal proofs remain first-class when relevant. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -432,8 +437,9 @@ def get_volume_organization_system_prompt() -> str: BODY CHAPTERS (from existing papers or gaps): - Select existing papers that directly contribute to answering the question -- Order them logically (foundations → main results → applications) -- Identify gaps: topics that need coverage but no paper exists +- Order them by the actual dependency structure of the solution +- Valid structures include mathematical foundations → results → proofs; constraints → mechanism → implementation → validation; hypothesis → evidence → protocol → limitations; or another justified sequence +- Identify gaps in proof, design, implementation, evidence, evaluation, validation, safety, risk, or another necessary answer component - Gap papers will be written before introduction/conclusion - Exclude chapters that are merely adjacent if they do not materially strengthen the answer @@ -494,55 +500,55 @@ def get_volume_organization_json_schema() -> str: - Body chapters (existing papers and gap papers) are in logical order - Conclusion is always the last chapter -EXAMPLE: +EXAMPLE (Mixed Engineering and Formal Analysis): { - "volume_title": "The Langlands Program: Connections Between Automorphic Forms and Galois Representations", + "volume_title": "A Resilient Water-Treatment System: Mechanism, Formal Limits, Implementation, and Validation", "chapters": [ { "chapter_type": "introduction", "paper_id": null, - "title": "Introduction: The Vision of the Langlands Program", + "title": "Introduction: The Solution and Its Evidence Boundaries", "order": 1, - "description": "Frames the user's question, provides historical context, and outlines the volume structure" + "description": "Frames the objective, the proposed system, the formal bounds, and the evidence status of each component" }, { "chapter_type": "existing_paper", "paper_id": "paper_003", - "title": "Automorphic Forms and Their Properties", + "title": "Core Treatment Mechanism and Constraints", "order": 2, - "description": "Foundational chapter covering automorphic forms" + "description": "Establishes the physical mechanism, operating constraints, and causal assumptions" }, { "chapter_type": "existing_paper", "paper_id": "paper_007", - "title": "Galois Representations in Number Theory", + "title": "Formal Performance and Safety Bounds", "order": 3, - "description": "Covers Galois representations and their role" + "description": "Provides mathematical bounds that supplement and constrain the engineering design" }, { "chapter_type": "gap_paper", "paper_id": null, - "title": "The Local Langlands Correspondence", + "title": "Prototype Implementation and Controlled Validation Protocol", "order": 4, - "description": "New paper needed to bridge automorphic forms and Galois representations locally" + "description": "Closes the implementation and empirical-validation gap without claiming unperformed tests" }, { "chapter_type": "existing_paper", "paper_id": "paper_015", - "title": "Applications and Computational Aspects", + "title": "Failure Modes, Monitoring, and Risk Mitigation", "order": 5, - "description": "Practical applications of the correspondence" + "description": "Analyzes operational risks and the controls needed for a credible deployment" }, { "chapter_type": "conclusion", "paper_id": null, - "title": "Conclusion: The Current State of the Langlands Program", + "title": "Conclusion: The Strongest Defensible Answer and Required Validation", "order": 6, - "description": "Synthesizes all chapters and directly answers the user's question about the connections" + "description": "Synthesizes the mechanism, formal bounds, implementation, evidence, and remaining validation" } ], "outline_complete": true, - "reasoning": "This volume uses three existing papers covering the major topics (automorphic forms, Galois representations, applications) and adds a gap paper on local Langlands correspondence which is essential but wasn't covered. The introduction and conclusion will frame and synthesize respectively, providing a complete answer to the user's question about the connections in the Langlands program." + "reasoning": "The answer has genuinely independent but dependent components: mechanism and constraints, formal bounds, implementation and validation, and risk controls. The gap paper is necessary to close the implementation and evidence gap; the volume does not add chapters merely because several papers exist." }""" @@ -561,7 +567,7 @@ def get_volume_validator_system_prompt() -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been proven correct. Apply domain- and claim-appropriate rigor, preserve the evidence status of every conclusion, and retain mathematical reasoning and formal proof as first-class modalities whenever relevant. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -575,11 +581,12 @@ def get_volume_validator_system_prompt() -> str: ACCEPT the organization if: - The volume title appropriately represents the answer - Existing papers are well-chosen and properly ordered -- Any gap papers identified are genuinely needed +- Any proof, design, implementation, evidence, evaluation, validation, safety, or risk gap papers are genuinely needed - Introduction and conclusion are properly planned - The reasoning is sound - If outline_complete=true, the structure is ready for writing - The structure stays focused on the strongest rigorous direct answer without unnecessary breadth +- Chapter order follows the actual dependency structure of the solution REJECT the organization if: - Important existing papers are missing @@ -648,7 +655,7 @@ def get_volume_intro_paper_context_prompt() -> str: YOUR TASK: Write an introduction that: 1. Clearly states the user's original research question -2. Provides historical and mathematical context +2. Provides the domain context, including mathematical foundations when relevant 3. Outlines the structure of the volume 4. Explains how each chapter contributes to answering the question 5. Sets expectations for what the reader will learn @@ -674,7 +681,7 @@ def get_volume_conclusion_paper_context_prompt() -> str: Write a conclusion that: 1. Synthesizes findings from ALL body chapters 2. DIRECTLY ANSWERS the user's original research question -3. States what is known WITH CERTAINTY (from the certainty assessment) +3. States the strongest defensible findings and preserves their evidence status 4. Acknowledges limitations and open questions 5. Provides a definitive take on the research question @@ -723,28 +730,28 @@ def build_certainty_assessment_prompt( if rejection_context: parts.append(f"{rejection_context}\n---\n") - # Add papers + # Always retain the complete library map; selected expansions supplement it. + parts.append("RESEARCH PAPERS (Abstracts and Outlines):\n") + for p in papers_summary: + parts.append(f"\n--- Paper ID: {p.get('paper_id', 'Unknown')} ---") + parts.append(f"\nTitle: {p.get('title', 'N/A')}") + parts.append(f"\nAbstract: {p.get('abstract', 'N/A')}") + if p.get('outline'): + parts.append(f"\nOutline:\n{p.get('outline')}") + parts.append(f"\nWord Count: {p.get('word_count', 0)}") + parts.append("\n") + if expanded_papers: - parts.append("RESEARCH PAPERS (Full Content):\n") + parts.append("\n---\nSELECTED FULL-CONTENT OR RETRIEVED EVIDENCE:\n") for p in expanded_papers: parts.append(f"\n{'=' * 60}") parts.append(f"\nPaper ID: {p.get('paper_id', 'Unknown')}") parts.append(f"\nTitle: {p.get('title', 'N/A')}") parts.append(f"\n{'=' * 60}") parts.append(f"\n\n{p.get('content', '[Content not available]')}\n") - else: - parts.append("RESEARCH PAPERS (Abstracts and Outlines):\n") - for p in papers_summary: - parts.append(f"\n--- Paper ID: {p.get('paper_id', 'Unknown')} ---") - parts.append(f"\nTitle: {p.get('title', 'N/A')}") - parts.append(f"\nAbstract: {p.get('abstract', 'N/A')}") - if p.get('outline'): - parts.append(f"\nOutline:\n{p.get('outline')}") - parts.append(f"\nWord Count: {p.get('word_count', 0)}") - parts.append("\n") parts.append("\n---\n") - parts.append("Assess what can be answered WITH CERTAINTY based on these papers (respond as JSON):") + parts.append("Assess the strongest defensible answer and the evidence status of its components based on these papers (respond as JSON):") return "".join(parts) @@ -752,7 +759,8 @@ def build_certainty_assessment_prompt( def build_certainty_validation_prompt( user_research_prompt: str, papers_summary: List[Dict[str, Any]], - assessment: Dict[str, Any] + assessment: Dict[str, Any], + expanded_papers: List[Dict[str, Any]] = None, ) -> str: """Build the certainty validation prompt.""" parts = [ @@ -762,16 +770,30 @@ def build_certainty_validation_prompt( "\n---\n", f"USER'S RESEARCH QUESTION:\n{user_research_prompt}", "\n---\n", - "RESEARCH PAPERS REVIEWED:\n" + "RESEARCH EVIDENCE CATALOG REVIEWED:\n" ] for p in papers_summary: - parts.append(f"- {p.get('paper_id')}: {p.get('title')}\n") + parts.append(f"\n--- Paper ID: {p.get('paper_id', 'Unknown')} ---") + parts.append(f"\nTitle: {p.get('title', 'N/A')}") + parts.append(f"\nAbstract: {p.get('abstract', 'N/A')}") + if p.get("outline"): + parts.append(f"\nOutline:\n{p.get('outline')}") + parts.append("\n") + + if expanded_papers: + parts.append("\n---\nSELECTED FULL-CONTENT OR RETRIEVED EVIDENCE:\n") + for p in expanded_papers: + parts.append(f"\n{'=' * 60}") + parts.append(f"\nPaper ID: {p.get('paper_id', 'Unknown')}") + parts.append(f"\nTitle: {p.get('title', 'N/A')}") + parts.append(f"\n{'=' * 60}") + parts.append(f"\n\n{p.get('content', '[Content not available]')}\n") parts.append("\n---\n") parts.append("CERTAINTY ASSESSMENT TO VALIDATE:\n") parts.append(f"Certainty Level: {assessment.get('certainty_level')}\n") - parts.append(f"Known Certainties: {assessment.get('known_certainties_summary')}\n") + parts.append(f"Defensible Findings and Evidence Status: {assessment.get('known_certainties_summary')}\n") parts.append(f"Reasoning: {assessment.get('reasoning')}\n") parts.append("\n---\n") parts.append("Validate this assessment (respond as JSON):") @@ -800,7 +822,7 @@ def build_format_selection_prompt( parts.append("CERTAINTY ASSESSMENT (Phase 1 Result):\n") parts.append(f"Certainty Level: {certainty_assessment.get('certainty_level')}\n") - parts.append(f"Known Certainties: {certainty_assessment.get('known_certainties_summary')}\n") + parts.append(f"Defensible Findings and Evidence Status: {certainty_assessment.get('known_certainties_summary')}\n") parts.append("\n---\n") parts.append("AVAILABLE RESEARCH PAPERS:\n") @@ -868,7 +890,7 @@ def build_volume_organization_prompt( parts.append(f"CERTAINTY ASSESSMENT:\n") parts.append(f"Certainty Level: {certainty_assessment.get('certainty_level')}\n") - parts.append(f"Known Certainties: {certainty_assessment.get('known_certainties_summary')}\n") + parts.append(f"Defensible Findings and Evidence Status: {certainty_assessment.get('known_certainties_summary')}\n") parts.append("\n---\n") parts.append("AVAILABLE PAPERS (can be used as body chapters):\n") @@ -951,7 +973,7 @@ def build_final_paper_title_prompt( parts.append("CERTAINTY ASSESSMENT:\n") parts.append(f"Certainty Level: {certainty_assessment.get('certainty_level')}\n") - parts.append(f"Known Certainties: {certainty_assessment.get('known_certainties_summary')}\n") + parts.append(f"Defensible Findings and Evidence Status: {certainty_assessment.get('known_certainties_summary')}\n") parts.append("\n---\n") parts.append("REFERENCE PAPERS (informing the answer):\n") diff --git a/backend/autonomous/prompts/paper_continuation_prompts.py b/backend/autonomous/prompts/paper_continuation_prompts.py index fccb8da..0c81879 100644 --- a/backend/autonomous/prompts/paper_continuation_prompts.py +++ b/backend/autonomous/prompts/paper_continuation_prompts.py @@ -8,7 +8,7 @@ def get_continuation_decision_system_prompt() -> str: """Get system prompt for brainstorm paper continuation decision.""" - return """You are an autonomous mathematical research agent deciding whether to write another paper from the current brainstorm or move on to a new research topic. Your role is to: + return """You are an autonomous research agent deciding whether to write another paper from the current brainstorm or move on to a new research topic. Your role is to: 1. Review the user's high-level research goal 2. Review the current brainstorm topic and its full database of accepted submissions @@ -26,7 +26,7 @@ def get_continuation_decision_system_prompt() -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been established as correct. Apply domain- and claim-appropriate rigor and honest provenance. Mathematical reasoning, theorem discovery, proof, and formalization remain first-class whenever relevant. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -45,10 +45,10 @@ def get_continuation_decision_system_prompt() -> str: WRITE ANOTHER PAPER if: - The brainstorm database contains substantial material not covered by existing paper(s) -- Another paper would address a meaningfully DIFFERENT angle, perspective, or subset of the brainstorm that improves direct resolution of the user's goal +- Another paper would develop a meaningfully DIFFERENT credible mechanism, proof, design, implementation route, evidence package, risk analysis, algorithm, or other answer-bearing contribution that improves direct resolution of the user's goal - The uncovered material is rich enough for a complete, distinct paper (not just leftover fragments) - Writing another paper from this brainstorm advances the user's goal MORE than starting a new topic -- The existing paper(s) focused on specific aspects, leaving other important aspects unexplored +- The existing paper(s) focused on specific aspects, leaving another substantial answer-bearing contribution unexplored - Another paper would more directly answer the user's prompt or a necessary piece of it MOVE ON if: @@ -87,19 +87,19 @@ def get_continuation_decision_json_schema() -> str: Write Another Paper: { "decision": "write_another_paper", - "reasoning": "The brainstorm database contains 22 submissions covering both algebraic and analytic approaches to the Langlands correspondence. Paper 1 focused exclusively on the algebraic side (Galois representations, class field theory). The analytic side (automorphic forms, L-functions, spectral theory) has substantial unexplored material in submissions 8, 12, 14, 17-20 that would form a distinct and valuable second paper." + "reasoning": "Paper 1 developed the desalination membrane mechanism, but submissions 8, 12, 14, and 17-20 contain a distinct evidence package: falsifiable degradation hypotheses, control conditions, safety risks, and a pilot validation protocol. That material can support a separate paper without claiming the proposed experiments have already been run." } Move On: { "decision": "move_on", - "reasoning": "The existing paper comprehensively covers the brainstorm's core content on modular forms and their connections to Galois representations. The remaining submissions (3 out of 18) contain supplementary remarks that are too fragmented for a standalone paper. The user's research goal on the Langlands program would be better served by exploring a new avenue such as trace formulas or p-adic methods." + "reasoning": "The existing paper already captures the brainstorm's protocol mechanism, constraints, failure modes, and verification plan. The remaining submissions are supplementary implementation notes rather than a distinct answer-bearing contribution, so a new topic would better advance the user's objective." }""" def get_continuation_validator_system_prompt() -> str: """Get system prompt for validating a continuation decision.""" - return """You are validating a brainstorm continuation decision in an autonomous mathematical research system. Your role is to: + return """You are validating a brainstorm continuation decision in an autonomous research system. Your role is to: 1. Review the user's high-level research goal 2. Review the current brainstorm topic and its database @@ -122,6 +122,8 @@ def get_continuation_validator_system_prompt() -> str: YOUR TASK: Validate whether the proposed continuation decision is the best use of research resources for improving the strongest rigorous direct answer. +Apply domain- and claim-appropriate rigor. A distinct mathematical proof is first-class when relevant. Empirical or engineering proposals must identify mechanisms, constraints, failure modes, evidence needs, and verification plans without pretending that proposed experiments, artifacts, or tests already exist. + ACCEPT the decision if: 1. WRITE_ANOTHER_PAPER: The brainstorm genuinely has enough distinct unexplored material for another paper AND the reasoning correctly identifies what material remains AND why it materially strengthens direct resolution 2. MOVE_ON: The existing papers adequately cover the brainstorm OR a new topic would genuinely better serve the goal AND the reasoning is sound @@ -168,7 +170,7 @@ def get_continuation_validator_json_schema() -> str: EXAMPLE (Accept): { "decision": "accept", - "reasoning": "The proposal to write another paper is well-justified. The brainstorm contains substantial analytic content (automorphic forms, L-functions) that paper 1's algebraic focus did not address. This material is rich enough for a distinct second paper." + "reasoning": "The proposal to write another paper is well-justified. Paper 1 covers the core mechanism, while the remaining brainstorm contains a distinct risk analysis and falsifiable validation package with explicit controls and failure criteria. This is substantial enough for a separate answer-bearing paper." } EXAMPLE (Reject - Use Structured Format): diff --git a/backend/autonomous/prompts/paper_redundancy_prompts.py b/backend/autonomous/prompts/paper_redundancy_prompts.py index d0ef056..7e8a000 100644 --- a/backend/autonomous/prompts/paper_redundancy_prompts.py +++ b/backend/autonomous/prompts/paper_redundancy_prompts.py @@ -24,7 +24,7 @@ def get_paper_redundancy_system_prompt() -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been proven correct. Apply the rigor and evidence standard appropriate to each domain and claim type. Mathematical claims require sound derivation or proof; empirical, artifact, engineering, software, and causal claims require corresponding evidence, provenance, feasibility reasoning, and validation. Use internal context as exploration history, not authority. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -45,16 +45,17 @@ def get_paper_redundancy_system_prompt() -> str: 2. OVERLAPS significantly with more comprehensive papers 3. Contains information SUPERSEDED by better, more complete papers 4. Was MARGINALLY useful initially but provides no unique value given current library -5. Covers the same mathematical territory as a newer, superior paper +5. Covers the same solution territory as a newer, superior paper 6. Is more indirect or auxiliary while another paper provides a stronger rigorous direct answer on the same territory REASONS TO KEEP - A paper should be kept if it: 1. Provides a stronger direct answer to the user's prompt than overlapping papers -2. Provides unique mathematical content that materially strengthens a direct route to the user's prompt +2. Provides a distinct solution mechanism or causal route 3. Offers a different perspective or approach that materially improves the strongest direct answer path -4. Contains specific proofs, theorems, or techniques necessary for direct prompt progress -5. Contributes to research diversity only when that diversity improves credible direct-answer progress -6. Covers distinct mathematical subtopics only when those subtopics are necessary to the user's prompt +4. Contributes distinct evidence, derivation, argument, implementation, or design necessary for direct prompt progress +5. Contains a distinct theorem, proof, algorithm, or impossibility result +6. Provides a distinct experimental proposal, validation method, risk, failure-mode, or limitation analysis +7. Contributes diversity only when that diversity improves credible direct-answer progress CONSERVATIVE APPROACH: - When in doubt, DO NOT recommend removal @@ -100,14 +101,14 @@ def get_paper_redundancy_json_schema() -> str: { "should_remove": true, "paper_id": "paper_005", - "reasoning": "Paper 005 on 'Basic Principles of Class Field Theory' is now redundant. Papers 003, 009, and 014 provide more comprehensive coverage of class field theory with deeper mathematical rigor and broader applications. Paper 005's unique contributions (elementary introduction) are minimal and the library would be stronger without this redundant entry. The other papers fully subsume its content." + "reasoning": "Paper 005 is now redundant. Papers 003, 009, and 014 provide the same mechanism and evidence with more complete feasibility analysis and validation. Paper 005 adds no distinct implementation, proof, experiment, risk analysis, or other direct-answer value, so the other papers fully subsume it." } No Removal: { "should_remove": false, "paper_id": null, - "reasoning": "After reviewing all paper titles and abstracts, no papers are redundant. Each paper provides unique perspectives, covers distinct mathematical areas, or approaches common topics from different angles. The library maintains good diversity without unnecessary overlap." + "reasoning": "After reviewing all paper titles and abstracts, no papers are redundant. Each paper contributes a distinct solution mechanism, evidence base, implementation, validation method, risk analysis, theorem, proof, or other necessary direct-answer component. The library maintains useful diversity without unnecessary overlap." }""" diff --git a/backend/autonomous/prompts/paper_reference_prompts.py b/backend/autonomous/prompts/paper_reference_prompts.py index 51ba212..11e7270 100644 --- a/backend/autonomous/prompts/paper_reference_prompts.py +++ b/backend/autonomous/prompts/paper_reference_prompts.py @@ -8,7 +8,7 @@ This is the CRUCIAL MECHANISM that enables COMPOUNDING KNOWLEDGE across research cycles. By selecting reference papers before brainstorming, submitters can: -- Build upon promising mathematical frameworks from prior AI-generated papers while independently re-checking their claims +- Build upon promising frameworks, results, proofs, mechanisms, designs, methods, evidence, algorithms, failures, or constraints while independently re-checking their claims - Avoid re-exploring territory already covered in depth - Identify novel connections between new topics and previously explored results - Accelerate convergence on valuable insights by standing on prior work @@ -43,7 +43,7 @@ def get_pre_brainstorm_expansion_system_prompt(max_papers: int) -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been established as correct. Use it only as exploration history and apply domain- and claim-appropriate verification. Mathematical reasoning and formal proof remain first-class whenever relevant. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -59,15 +59,15 @@ def get_pre_brainstorm_expansion_system_prompt(max_papers: int) -> str: WHY THIS MATTERS - COMPOUNDING KNOWLEDGE: This is the crucial mechanism that allows the system to compound knowledge across research cycles. By selecting reference papers BEFORE brainstorming, you can: -- Build upon promising mathematical frameworks from prior AI-generated papers, while independently re-checking their claims +- Build upon promising frameworks, results, proofs, mechanisms, designs, methods, evidence, algorithms, failures, or constraints from prior AI-generated papers, while independently re-checking their claims - Avoid re-exploring territory already covered in depth - Identify novel connections between your new topic and previously explored results - Accelerate convergence on valuable insights by standing on prior work THRESHOLD: "VERY USEFUL FOR BRAINSTORMING" -- Papers that provide mathematical foundations you'll directly build upon -- Papers that cover related concepts you can extend or connect to in service of a more direct answer -- Papers that offer techniques or methods that materially strengthen the most direct route to your topic +- Papers that provide credible frameworks, results, proofs, mechanisms, designs, methods, evidence, algorithms, failures, or constraints you'll directly build upon +- Papers that cover related concepts or mathematical foundations you can extend or connect to in service of a more direct answer +- Papers that offer techniques or methods that materially strengthen the most direct route to your topic, with provenance and uncertainty treated honestly - Don't request papers that are merely tangentially related OPTIONS: @@ -109,7 +109,7 @@ def get_additional_reference_expansion_system_prompt(max_total_papers: int) -> s - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been established as correct. Use it only as exploration history and apply domain- and claim-appropriate verification. Mathematical reasoning and formal proof remain first-class whenever relevant. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -129,7 +129,7 @@ def get_additional_reference_expansion_system_prompt(max_total_papers: int) -> s THRESHOLD: "VALUABLE BASED ON BRAINSTORM INSIGHTS" - Papers that address topics that emerged during brainstorming and materially strengthen direct resolution -- Papers that provide additional techniques you now realize are relevant to the strongest direct answer +- Papers that provide additional mechanisms, designs, methods, evidence, algorithms, proofs, constraints, or failure analyses you now realize are relevant to the strongest direct answer - Papers that cover connections you discovered during exploration only when those connections improve direct progress - Don't add papers just to fill slots @@ -152,7 +152,7 @@ def get_additional_reference_expansion_system_prompt(max_total_papers: int) -> s def get_reference_expansion_system_prompt(max_papers: int = 6) -> str: """Get system prompt for reference expansion request (Step 1: abstracts only).""" - return f"""You are selecting reference papers for an upcoming mathematical research paper. Your role is to: + return f"""You are selecting reference papers for an upcoming solution-oriented research paper or report. Your role is to: 1. Review your brainstorm topic and database 2. Review titles and abstracts of existing papers in the library @@ -169,7 +169,7 @@ def get_reference_expansion_system_prompt(max_papers: int = 6) -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been established as correct. Use it only as exploration history and apply domain- and claim-appropriate verification. Mathematical reasoning and formal proof remain first-class whenever relevant. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -183,7 +183,7 @@ def get_reference_expansion_system_prompt(max_papers: int = 6) -> str: - Do not expand papers that are merely adjacent background unless they are needed for direct resolution THRESHOLD: "VERY USEFUL" -- A paper is "very useful" if it provides substantial mathematical context, techniques, or insights that materially strengthen the most direct answer to your brainstorm topic +- A paper is "very useful" if it provides substantial credible context, results, proofs, mechanisms, designs, methods, evidence, algorithms, failures, constraints, or insights that materially strengthen the most direct answer to your brainstorm topic - Don't request papers that are merely tangentially related - Quality over quantity - only request papers you genuinely need to evaluate @@ -224,20 +224,20 @@ def get_reference_expansion_json_schema() -> str: { "expand_papers": ["paper_003", "paper_007", "paper_011"], "proceed_without_references": false, - "reasoning": "Papers 003, 007, and 011 appear highly relevant based on their abstracts. Paper 003 covers class field theory which connects directly to our brainstorm on reciprocity laws. Papers 007 and 011 discuss Galois representations and modular forms respectively, both central to our upcoming paper. Need to see full content to assess their utility for reference." + "reasoning": "Papers 003, 007, and 011 appear highly relevant based on their abstracts. Paper 003 proposes a membrane-cleaning mechanism, paper 007 analyzes energy and materials constraints, and paper 011 defines a falsifiable pilot protocol. Their full content is needed to assess mechanisms, evidence provenance, failure modes, and direct value." } Proceed Without References: { "expand_papers": [], "proceed_without_references": true, - "reasoning": "After reviewing all existing paper abstracts, none meet the 'very useful' threshold for the upcoming paper on modular forms and Galois representations. The existing papers focus on different aspects of Langlands program (L-functions, automorphic forms) that don't provide direct reference value for this specific paper topic." + "reasoning": "After reviewing all existing abstracts, none meet the 'very useful' threshold for the upcoming fault-tolerant protocol report. They concern unrelated application domains and provide no mechanism, proof, evidence, algorithm, or constraint that materially improves this objective." }""" def get_reference_selection_system_prompt(max_papers: int) -> str: """Get system prompt for final reference selection (Step 2: full papers).""" - return f"""You are making your FINAL SELECTION of reference papers for an upcoming mathematical research paper. Your role is to: + return f"""You are making your FINAL SELECTION of reference papers for an upcoming solution-oriented research paper or report. Your role is to: 1. Review your brainstorm topic and database 2. Review the FULL CONTENT of the papers you requested to expand @@ -254,7 +254,7 @@ def get_reference_selection_system_prompt(max_papers: int) -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been established as correct. Use it only as exploration history and apply domain- and claim-appropriate verification. Mathematical reasoning and formal proof remain first-class whenever relevant. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -268,10 +268,10 @@ def get_reference_selection_system_prompt(max_papers: int) -> str: - Prefer papers that directly strengthen the answer over broader background SELECTION CRITERIA: -- Papers that provide essential mathematical background for the direct answer -- Papers that offer techniques or methods central to your topic's strongest resolution path -- Papers that establish theoretical foundations you'll directly build upon -- Papers that present related results you'll reference or extend in order to answer the question more directly +- Papers that provide essential credible background, evidence, or mathematical foundations for the direct answer +- Papers that offer mechanisms, designs, techniques, methods, or algorithms central to your topic's strongest resolution path +- Papers that establish results, proofs, constraints, or failure analyses you'll directly build upon +- Papers whose claims can be independently checked and whose provenance and limitations support honest synthesis CONSTRAINT: - Maximum {max_papers} papers can be selected (hard limit for context budget) @@ -301,7 +301,7 @@ def get_reference_selection_json_schema(max_papers: int) -> str: EXAMPLE: {{ "selected_papers": ["paper_003", "paper_007", "paper_011"], - "reasoning": "After reviewing full content, these three papers provide the most useful reference material: Paper 003 establishes the class field theory foundation needed for our reciprocity discussions. Paper 007's treatment of Galois representations will inform our theoretical sections. Paper 011's computational examples of modular forms will enhance our practical demonstrations. The other expanded papers, while relevant, overlap too much with our brainstorm content or cover tangential topics." + "reasoning": "After reviewing full content, these papers provide the most useful reference material: Paper 003 supplies the core mechanism, Paper 007 makes the governing constraints explicit, and Paper 011 provides a falsifiable validation method without claiming completed results. The other expanded papers are tangential or duplicate the brainstorm." }}""" diff --git a/backend/autonomous/prompts/paper_title_exploration_prompts.py b/backend/autonomous/prompts/paper_title_exploration_prompts.py index a5b7f90..1520a9d 100644 --- a/backend/autonomous/prompts/paper_title_exploration_prompts.py +++ b/backend/autonomous/prompts/paper_title_exploration_prompts.py @@ -50,10 +50,12 @@ def build_title_exploration_user_prompt( parts.append("- Titles too similar to already-completed related papers should be rejected") parts.append("- The goal is to map multiple plausible title directions before committing\n") parts.append("WHAT MAKES A GOOD CANDIDATE TITLE:") - parts.append("- Accurately captures the paper's likely mathematical content") + parts.append("- Accurately captures the paper's likely solution contribution and actual scope") parts.append("- Specific enough to communicate the core focus") parts.append("- Foregrounds the direct answer, core result, or limitation when justified") - parts.append("- Professional and suitable for a mathematical research paper") + parts.append("- Professional and suitable for a solution-oriented research paper or report") + parts.append("- Uses mathematical title forms when the work is mathematical; mathematics, theorems, and proofs remain first-class when relevant") + parts.append("- Does not imply completed experiments, measurements, implementations, or validation when the source contains only proposals") parts.append("- Distinct from already-accepted candidate titles") parts.append("- Distinct from related completed papers listed below") parts.append("- If this is a final-answer or chapter paper, the title should match that role directly\n") diff --git a/backend/autonomous/prompts/paper_title_prompts.py b/backend/autonomous/prompts/paper_title_prompts.py index d1d9f6e..f84225a 100644 --- a/backend/autonomous/prompts/paper_title_prompts.py +++ b/backend/autonomous/prompts/paper_title_prompts.py @@ -8,7 +8,7 @@ def get_paper_title_system_prompt() -> str: """Get system prompt for paper title selection.""" - return """You are selecting a title for a mathematical research paper. Your role is to: + return """You are selecting a title for a solution-oriented research paper or report. Your role is to: 1. Review your brainstorm topic and database content 2. Review any selected reference papers informing this paper (if any) @@ -26,14 +26,14 @@ def get_paper_title_system_prompt() -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been established as correct. Use domain- and claim-appropriate rigor and honest provenance. Mathematical reasoning, theorem discovery, proof, and formalization remain first-class whenever relevant. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. --- YOUR TASK: -Choose a title that accurately captures the mathematical content and scope of the planned paper. +Choose a title that accurately captures the strongest credible solution contribution and scope of the planned paper. DIRECT-SOLUTION PREFERENCE: - When the paper reaches direct answer-bearing content, let the title foreground that content rather than sounding like generic exploration @@ -46,18 +46,19 @@ def get_paper_title_system_prompt() -> str: - If "EXISTING PAPERS FROM THIS BRAINSTORM: None" - there's nothing to differentiate from TITLE CRITERIA: -- Accurately represents the mathematical content to be covered (from your brainstorm) +- Accurately represents the solution content to be covered (from your brainstorm) - Is specific enough to convey the paper's focus -- Is professional and suitable for a mathematical research paper +- Is professional and suitable for a solution-oriented research paper or report - Differentiates from EXISTING COMPLETED PAPERS from the same brainstorm (if any exist - check the list below) - Avoids being overly broad or generic - Makes the paper's strongest direct contribution clear when the content justifies it TITLE STYLE: -- Use standard mathematical paper title conventions +- Use conventions appropriate to the domain and contribution; retain standard mathematical title forms when the paper is mathematical - Can include colons for subtitles if appropriate -- Should convey the main mathematical themes -- Consider including key mathematical objects/concepts +- Should convey the main mechanism, result, design, method, evidence package, limitation, algorithm, proof, or other answer-bearing contribution +- Consider including key mathematical objects/concepts when relevant +- Do not imply that proposed experiments, artifacts, measurements, or validation already exist - Appropriate length: typically 5-15 words CRITICAL JSON ESCAPE RULES: @@ -73,7 +74,7 @@ def get_paper_title_json_schema() -> str: """Get JSON schema for paper title selection.""" return """REQUIRED JSON FORMAT: { - "paper_title": "string - The complete title for the mathematical research paper", + "paper_title": "string - The complete title for the solution-oriented research paper or report", "reasoning": "string - Why this title appropriately captures the brainstorm content and differentiates from existing papers (if any)" } @@ -83,8 +84,8 @@ def get_paper_title_json_schema() -> str: EXAMPLE: { - "paper_title": "Modular Forms and Galois Representations in the Langlands Program: A Computational Perspective", - "reasoning": "This title accurately captures the core content of our brainstorm database, which extensively covers both modular forms and Galois representations with emphasis on computational approaches. The subtitle 'A Computational Perspective' differentiates it from the existing theoretical paper on Langlands correspondence in our library and reflects the practical examples and algorithms present in the brainstorm submissions." + "paper_title": "Partition-Tolerant Coordination with Explicit Safety, Liveness, and Recovery Constraints", + "reasoning": "This title identifies the report's actual protocol contribution and its verification obligations. It reflects the source brainstorm while differing from the existing completed paper on failure detection, and it does not imply that the proposed implementation or tests have already been completed." }""" @@ -109,7 +110,7 @@ def get_paper_title_validator_system_prompt() -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been established as correct. Apply domain- and claim-appropriate rigor and honest provenance. Mathematical reasoning and formal proof remain first-class whenever relevant. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -141,18 +142,20 @@ def get_paper_title_validator_system_prompt() -> str: - It remains consistent with the paper's intended scope when selected reference papers are present - It is appropriately specific (not too broad or narrow) - It differentiates from EXISTING COMPLETED PAPERS from the same brainstorm (if any exist) -- It follows mathematical paper title conventions +- It follows professional conventions appropriate to the paper's domain; mathematical papers may and should use mathematical conventions - The reasoning is sound - If "EXISTING PAPERS FROM THIS BRAINSTORM: None" - there's nothing to differentiate from, so accept if other criteria are met - It makes any justified direct conclusion or core result clear rather than sounding needlessly exploratory +- It does not overclaim completed empirical demonstration, artifacts, measurements, citations, or validation beyond the source material REJECT the title if: - It is too similar to an EXISTING COMPLETED PAPER from the same brainstorm (NOT brainstorm submissions - those are the source material!) - It doesn't accurately represent the brainstorm content - It is too vague or generic -- It doesn't follow professional conventions +- It doesn't follow professional conventions appropriate to the domain - The reasoning is flawed - It obscures a clear direct result behind generic exploratory wording +- It overstates a proposal or hypothesis as an implemented, tested, measured, or demonstrated result DO NOT REJECT simply because the title reflects brainstorm submission content - that is the INTENDED behavior. diff --git a/backend/autonomous/prompts/proof_prompts.py b/backend/autonomous/prompts/proof_prompts.py index dc8d390..8522695 100644 --- a/backend/autonomous/prompts/proof_prompts.py +++ b/backend/autonomous/prompts/proof_prompts.py @@ -776,7 +776,6 @@ def build_proof_novelty_prompt( ) -> str: """Ask the validator to classify a Lean-verified theorem into one of five novelty tiers.""" user_prompt, _verified_proof_context_block = _prepare_user_prompt_context(user_prompt) - existing_proofs_block = existing_novel_proofs or "[No previously stored novel proofs.]" return f"""This proof has been FORMALLY VERIFIED by Lean 4. It is mathematically valid. Your ONLY task: assign a novelty tier to the verified result based on the criteria below. @@ -788,7 +787,6 @@ def build_proof_novelty_prompt( - It is a trivial identity, tautology, or definitional equality. - It is closable by a single standard tactic (simp, omega, norm_num, decide, rfl). - It is a routine helper lemma, proof-engineering fact, or general known-knowledge-base entry rather than new prompt-directed knowledge. -- It duplicates a result already present in the stored proofs below. - Assign this tier when there is no meaningful original contribution. "novel_formulation" @@ -832,9 +830,6 @@ def build_proof_novelty_prompt( LEAN 4 CODE: {lean_code} -EXISTING STORED NOVEL PROOFS: -{existing_proofs_block} - {_json_only_footer('{"novelty_tier": "mathematical_discovery", "reasoning": "brief explanation"}')} """ diff --git a/backend/autonomous/prompts/topic_exploration_prompts.py b/backend/autonomous/prompts/topic_exploration_prompts.py index b51b47d..2325ba9 100644 --- a/backend/autonomous/prompts/topic_exploration_prompts.py +++ b/backend/autonomous/prompts/topic_exploration_prompts.py @@ -31,7 +31,7 @@ def build_exploration_user_prompt( parts.append("=== TOPIC EXPLORATION PHASE ===\n") parts.append("You are in a TOPIC EXPLORATION phase. Your task is to propose CANDIDATE BRAINSTORM QUESTIONS") - parts.append("that maximize the chance of a rigorous DIRECT answer to the research goal below.\n") + parts.append("that maximize the chance of the strongest credible, genuinely novel, and rigorous DIRECT solution to the research goal below.\n") parts.append("First prefer candidate questions that aggressively attack the user's WHOLE question as stated,") parts.append("no partial solutions. If the whole question cannot be attacked in one shot,") parts.append("propose the next best necessary piece whose answer") @@ -43,13 +43,15 @@ def build_exploration_user_prompt( parts.append("WHAT MAKES A GOOD CANDIDATE QUESTION:") parts.append("- Most directly targets answering the user's whole problem") parts.append("- If piecewise, targets a clearly necessary piece of the full problem") - parts.append("- Specific enough to guide focused mathematical exploration (not vague)") + parts.append("- Specific enough to guide focused solution seeking (not vague)") parts.append("- Novel relative to already-accepted candidates and existing brainstorms") parts.append("- Relevant to the research goal below") - parts.append("- Opens a DISTINCT mathematical direction not already represented") + parts.append("- Opens a DISTINCT high-impact solution, invention, or research direction not already represented") parts.append("- Does not retreat to an easier adjacent/practical/background route when a direct whole-question route is available") - parts.append("- Grounded in established mathematical concepts") - parts.append("- Actionable — a brainstorm session could produce meaningful insights from it\n") + parts.append("- Identifies a concrete mechanism, decisive question, obstruction, design route, algorithm, theorem, experiment, or other problem-appropriate path") + parts.append("- Uses domain- and claim-appropriate rigor: mathematics, theorem discovery, proof, and formalization remain first-class whenever relevant") + parts.append("- For empirical, artifact, or literature-dependent routes, requires honest evidence/provenance or explicit hypothesis, proposed-test, or proposed-work language") + parts.append("- Actionable — a brainstorm session could produce meaningful progress from it\n") parts.append("DIVERSITY IS PARAMOUNT:") parts.append("Your candidate MUST be SUBSTANTIVELY DIFFERENT from already-accepted candidates.") parts.append("The goal is to compare the BEST direct-answer paths before committing to one.") diff --git a/backend/autonomous/prompts/topic_prompts.py b/backend/autonomous/prompts/topic_prompts.py index 4a00f11..1c1a635 100644 --- a/backend/autonomous/prompts/topic_prompts.py +++ b/backend/autonomous/prompts/topic_prompts.py @@ -6,12 +6,12 @@ def get_topic_selection_system_prompt() -> str: """Get system prompt for topic selection submitter.""" - return """You are an autonomous mathematical research agent selecting the next research avenue to explore. Your role is to: + return """You are an autonomous research agent selecting the next solution or research avenue to explore. Your role is to: 1. Review the user's high-level research goal 2. Review all existing brainstorm topics and their status 3. Review all completed papers (titles, abstracts, word counts) -4. Decide the best next action: start a new topic, continue an existing topic, or combine topics +4. Decide the best next action: start a new topic or continue an existing topic ⚠️ CRITICAL - INTERNAL CONTENT WARNING ⚠️ @@ -24,14 +24,14 @@ def get_topic_selection_system_prompt() -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been established as correct. Your role is to pursue rigorous, verifiable progress using the solution form and verification standard appropriate to the objective. Mathematical reasoning, theorem discovery, proof, and formalization remain first-class whenever relevant. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. --- YOUR TASK: -Select the optimal research avenue that most directly advances the user's research goal toward a rigorous answer. +Aggressively pursue the strongest credible and genuinely novel solution to the user's exact objective. Select the optimal problem-solving avenue, mechanism, design space, theorem route, experiment, algorithm, or research direction that most directly advances that objective. DIRECT-SOLUTION PREFERENCE: - First prefer avenues that aggressively attack the user's WHOLE question as stated, no partial solutions @@ -43,31 +43,27 @@ def get_topic_selection_system_prompt() -> str: DECISION OPTIONS: 1. NEW_TOPIC - Create a brand new brainstorm topic to explore 2. CONTINUE_EXISTING - Resume work on an incomplete brainstorm that has more value to explore -3. COMBINE_TOPICS - Merge multiple related brainstorms into a unified exploration DECISION CRITERIA: When to choose NEW_TOPIC: - All existing topics are complete OR -- A genuinely new mathematical avenue would provide more direct-answer value than continuing existing work +- A genuinely new solution or research avenue would provide more direct-answer value than continuing existing work - The new topic addresses an unexplored area relevant to the research goal -- Existing papers don't adequately cover this mathematical territory +- Existing papers don't adequately cover this problem-solving territory - The new topic offers a stronger direct route to resolving the user's whole question than current options When to choose CONTINUE_EXISTING: -- An incomplete brainstorm has significant untapped mathematical depth -- The brainstorm has few submissions relative to its mathematical richness +- An incomplete brainstorm has significant untapped solution value +- The brainstorm has few submissions relative to the richness of its mechanisms, constraints, evidence needs, designs, algorithms, proofs, or other relevant routes - Continuing would yield more valuable direct progress than starting fresh - The unfinished topic still contains a realistic path to a stronger direct answer to the whole prompt or a necessary piece of it -When to choose COMBINE_TOPICS: -- Multiple existing brainstorms are deeply interconnected -- A unified exploration would reveal insights neither topic could provide alone -- The mathematical concepts naturally bridge multiple brainstorms -- The combination produces a more direct route to answering the user's whole question than keeping them separate - CRITICAL REQUIREMENTS: -- Focus on mathematical rigor and logical soundness +- Apply domain- and claim-appropriate rigor, logical soundness, provenance, and honest uncertainty +- Mathematical claims require sound derivation, proof, or explicit assumptions; mathematics and formal proof remain first-class when relevant +- Empirical claims require actual evidence or explicit hypothesis/proposed-test language; artifact and literature claims must not invent implementations, tests, measurements, or citations +- Engineering, software, strategic, and causal routes require concrete mechanisms, constraints, failure modes, feasibility reasoning, and verification plans as appropriate - Avoid redundancy with existing work - Ensure topic selection serves the user's research goal - Consider the existing paper library to avoid redundant explorations @@ -89,18 +85,16 @@ def get_topic_selection_json_schema() -> str: """Get JSON schema for topic selection.""" return """REQUIRED JSON FORMAT: { - "action": "new_topic | continue_existing | combine_topics", + "action": "new_topic | continue_existing", "topic_id": "string - Required if action is continue_existing (e.g., 'topic_003')", - "topic_ids": ["array of topic_ids - Required if action is combine_topics (e.g., ['topic_001', 'topic_002'])"], - "topic_prompt": "string - Required if action is new_topic or combine_topics. The brainstorm question/avenue to explore", + "topic_prompt": "string - Required if action is new_topic. The brainstorm question/avenue to explore", "reasoning": "string - Why this is the best choice right now" } FIELD REQUIREMENTS: -- action: MUST be one of: "new_topic", "continue_existing", "combine_topics" +- action: MUST be one of: "new_topic", "continue_existing" - topic_id: Required ONLY if action is "continue_existing" -- topic_ids: Required ONLY if action is "combine_topics" (array of 2+ topic IDs) -- topic_prompt: Required if action is "new_topic" OR "combine_topics" +- topic_prompt: Required if action is "new_topic" - reasoning: ALWAYS required EXAMPLES: @@ -108,8 +102,8 @@ def get_topic_selection_json_schema() -> str: New Topic: { "action": "new_topic", - "topic_prompt": "Attack the most direct route toward the target Langlands correspondence rather than surveying adjacent background", - "reasoning": "This topic prioritizes the user's full Langlands goal. If the full correspondence cannot be resolved in one brainstorm, it asks for the ASI's best necessary next piece rather than a broad or easier detour." + "topic_prompt": "Design a fault-tolerant coordination mechanism that directly preserves safety and liveness during network partitions, with explicit failure assumptions and a validation plan", + "reasoning": "This topic directly attacks the user's distributed-protocol objective through a concrete mechanism and verification route rather than retreating to a broad survey of consensus systems." } Continue Existing: @@ -117,20 +111,12 @@ def get_topic_selection_json_schema() -> str: "action": "continue_existing", "topic_id": "topic_003", "reasoning": "The brainstorm on reciprocity laws has only 7 submissions and has not yet covered explicit formulas or computational approaches. Continuing this topic will provide more complete understanding before moving to a new avenue." -} - -Combine Topics: -{ - "action": "combine_topics", - "topic_ids": ["topic_002", "topic_005"], - "topic_prompt": "Combine the existing topics only insofar as their union creates a more direct route toward the user's full Langlands goal", - "reasoning": "Topics 002 and 005 are only worth combining if the combined route directly serves the user's full prompt. The merged topic should not become a broad survey of related theory." }""" def get_topic_validator_system_prompt() -> str: """Get system prompt for topic validator.""" - return """You are validating a topic selection decision in an autonomous mathematical research system. Your role is to: + return """You are validating a topic selection decision in an autonomous research system. Your role is to: 1. Review the user's high-level research goal 2. Review all existing brainstorm topics and their status @@ -148,38 +134,36 @@ def get_topic_validator_system_prompt() -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been established as correct. Judge it skeptically using domain- and claim-appropriate rigor. Mathematical reasoning and formal proof remain first-class whenever relevant, but non-mathematical work is not deficient merely for using the correct non-mathematical solution form. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. --- YOUR TASK: -Validate whether the proposed topic selection represents the best use of research resources for obtaining the strongest rigorous direct answer. +Validate whether the proposed topic selection represents the best use of research resources for obtaining the strongest credible, genuinely novel, rigorous direct solution to the user's exact objective. VALIDATION CRITERIA: ACCEPT the topic selection if: -1. NEW_TOPIC: The new topic addresses a genuinely valuable mathematical avenue not yet covered -2. CONTINUE_EXISTING: The brainstorm's current state justifies continuation (incomplete, mathematically rich) -3. COMBINE_TOPICS: There are clear mathematical connections that justify unification -4. The choice is relevant to the user's research goal -5. The reasoning is sound and mathematically grounded -6. The topic doesn't duplicate existing completed work -7. The choice aggressively addresses the user's whole question where possible -8. If it is piecewise, the piece is clearly necessary for progress on the full question -9. The choice is at least as direct a route to answering the user's question as the available alternatives +1. NEW_TOPIC: The new topic addresses a genuinely valuable solution or research avenue not yet covered +2. CONTINUE_EXISTING: The brainstorm's current state justifies continuation (incomplete, rich in answer-bearing work) +3. The choice is relevant to the user's research goal +4. The reasoning is domain-grounded, evidence-aware, logically sound, and honest about uncertainty +5. The topic doesn't duplicate existing completed work +6. The choice aggressively addresses the user's whole question where possible +7. If it is piecewise, the piece is clearly necessary for progress on the full question +8. The choice is at least as direct a route to answering the user's question as the available alternatives REJECT the topic selection if: 1. NEW_TOPIC: The topic duplicates an existing brainstorm or completed paper 2. CONTINUE_EXISTING: The brainstorm should be marked complete (exhausted) or a new topic would be more valuable -3. COMBINE_TOPICS: The topics lack clear mathematical connections for unification -4. The choice ignores more valuable research avenues -5. The reasoning is flawed or lacks mathematical rigor -6. The selection would lead to redundant work -7. It retreats to an easier adjacent/practical/background route while a direct whole-question attack is available -8. It proposes a piecewise topic without showing why that piece is necessary for solving the full user question -9. A clearly more direct rigorous avenue was available and unjustifiably ignored +3. The choice ignores more valuable research avenues +4. The reasoning is flawed, lacks claim-appropriate rigor, fabricates evidence/artifacts/sources, or omits relevant constraints and failure modes +5. The selection would lead to redundant work +6. It retreats to an easier adjacent/practical/background route while a direct whole-question attack is available +7. It proposes a piecewise topic without showing why that piece is necessary for solving the full user question +8. A clearly more direct rigorous avenue was available and unjustifiably ignored REJECTION FEEDBACK FORMAT: If rejecting, provide CONCRETE, ACTIONABLE guidance: @@ -215,7 +199,7 @@ def get_topic_validator_json_schema() -> str: EXAMPLE (Accept): { "decision": "accept", - "reasoning": "The proposed topic on modular forms represents a valuable new avenue that complements existing brainstorms on L-functions. The submitter correctly identifies this as a concrete entry point to Langlands correspondence that hasn't been explored in prior topics." + "reasoning": "The proposed topic develops a distinct low-energy membrane-cleaning mechanism with explicit constraints, measurable hypotheses, and a validation plan. It directly serves the desalination objective without claiming that the proposed experiment has already demonstrated the result." } EXAMPLE (Reject - Use Structured Format): @@ -262,9 +246,8 @@ def build_topic_selection_prompt( You may: - Select one of these candidates directly as your topic (action: new_topic, topic_prompt: the candidate question) -- Combine or synthesize multiple candidates into a stronger question +- Synthesize lessons from multiple candidates into a stronger question - Continue an existing brainstorm if the candidates reveal it is worth continuing -- Combine existing brainstorms if the candidates reveal connections - Propose something entirely new if the candidates missed a critical avenue {candidate_questions} @@ -366,8 +349,6 @@ def build_topic_validation_prompt( parts.append(f"Action: {proposed_action.get('action', 'Unknown')}") if proposed_action.get('topic_id'): parts.append(f"\nTopic ID: {proposed_action.get('topic_id')}") - if proposed_action.get('topic_ids'): - parts.append(f"\nTopic IDs to Combine: {', '.join(proposed_action.get('topic_ids', []))}") if proposed_action.get('topic_prompt'): parts.append(f"\nTopic Prompt: {proposed_action.get('topic_prompt')}") parts.append(f"\nReasoning: {proposed_action.get('reasoning', 'N/A')}") diff --git a/backend/autonomous/validation/paper_redundancy_checker.py b/backend/autonomous/validation/paper_redundancy_checker.py index e10a8a2..8ce0167 100644 --- a/backend/autonomous/validation/paper_redundancy_checker.py +++ b/backend/autonomous/validation/paper_redundancy_checker.py @@ -80,6 +80,10 @@ async def check_redundancy( user_research_prompt=user_research_prompt, papers_summary=papers_summary ) + from backend.shared.solution_path.integration import with_validator_hook + prompt = with_validator_hook( + prompt, getattr(self, "solution_path_manager", None) + ) # Generate task ID for tracking task_id = self.get_current_task_id() @@ -114,7 +118,7 @@ async def check_redundancy( # Parse JSON using central utility try: data = parse_json(content) - + from backend.shared.solution_path.integration import enqueue_optional_update should_remove = data.get("should_remove", False) paper_id = data.get("paper_id") reasoning = data.get("reasoning", "No reasoning provided") @@ -130,6 +134,15 @@ async def check_redundancy( if paper_id not in valid_ids: logger.warning(f"PaperRedundancyChecker: Invalid paper_id: {paper_id}") return self._create_no_removal(f"Invalid paper_id: {paper_id}") + + await enqueue_optional_update( + data, + getattr(self, "solution_path_manager", None), + proposer_role=self.role_id, + source_task_id=task_id, + source_phase="paper_redundancy_review", + source_decision="reject" if should_remove else "accept", + ) result = PaperRedundancyReviewResult( should_remove=should_remove, diff --git a/backend/compiler/agents/high_param_submitter.py b/backend/compiler/agents/high_param_submitter.py index ff7b5e7..5e42496 100644 --- a/backend/compiler/agents/high_param_submitter.py +++ b/backend/compiler/agents/high_param_submitter.py @@ -842,6 +842,12 @@ async def _step_discovery(self) -> Optional[dict]: source_material_context=source_material_context, source_material_label=self._source_material_label, ) + from backend.shared.solution_path.integration import with_budgeted_solver_plan + prompt = with_budgeted_solver_plan( + prompt, + getattr(self, "solution_path_manager", None), + self.available_input_tokens, + ) if count_tokens(prompt) > max_allowed: logger.warning("Rigor discovery prompt too large; retrying without RAG evidence") @@ -978,9 +984,14 @@ async def _on_attempt_feedback(feedback: ProofAttemptFeedback) -> None: return None if not success: - last_error = attempts[-1].error_output if attempts else "" + last_feedback = attempts[-1] if attempts else None + last_error = last_feedback.error_output if last_feedback else "" if MANDATORY_FULL_SOURCE_CONTEXT_OVERFLOW_PREFIX.lower() in last_error.lower(): - raise ValueError(last_error) + from backend.autonomous.agents.proof_formalization_agent import ( + ProofFormalizationContextOverflowError, + ) + + raise ProofFormalizationContextOverflowError(last_feedback) # Record as an open proof target so the next rigor cycle's # discovery step can optionally retry it. try: @@ -1205,6 +1216,12 @@ async def _build_placement_submission( placement_attempt=placement_attempt, validator_rejection_feedback=validator_rejection_feedback, ) + from backend.shared.solution_path.integration import with_budgeted_solver_plan + prompt = with_budgeted_solver_plan( + prompt, + getattr(self, "solution_path_manager", None), + self.available_input_tokens, + ) max_allowed = rag_config.get_available_input_tokens( self.context_window, self.max_output_tokens diff --git a/backend/compiler/agents/writer_submitter.py b/backend/compiler/agents/writer_submitter.py index e763ee0..dcfb089 100644 --- a/backend/compiler/agents/writer_submitter.py +++ b/backend/compiler/agents/writer_submitter.py @@ -47,6 +47,10 @@ WOLFRAM_MAX_CALLS_PER_SUBMISSION = 20 +class WolframFinalizationError(RuntimeError): + """Raised when a tool-using model will not produce final JSON after the cap.""" + + def _hash_text_for_audit(value: str) -> str: text = value or "" return hashlib.sha256(text.encode("utf-8", errors="replace")).hexdigest() if text else "" @@ -261,6 +265,14 @@ async def submit_outline_create(self) -> CompilerSubmission: rag_evidence=context_pack.text ) logger.info(f"Prompt built: {len(prompt)} chars") + from backend.shared.solution_path.integration import with_budgeted_solver_plan + prompt = with_budgeted_solver_plan( + prompt, + getattr(self, "solution_path_manager", None), + rag_config.get_available_input_tokens( + self.context_window, self.max_output_tokens + ), + ) task_id = self.get_current_task_id() await api_client_manager.prewarm_assistant_memory_context( @@ -416,6 +428,14 @@ async def submit_outline_update(self) -> Optional[CompilerSubmission]: rag_evidence=context_pack.text ) logger.info(f"Prompt built: {len(prompt)} chars") + from backend.shared.solution_path.integration import with_budgeted_solver_plan + prompt = with_budgeted_solver_plan( + prompt, + getattr(self, "solution_path_manager", None), + rag_config.get_available_input_tokens( + self.context_window, self.max_output_tokens + ), + ) task_id = self.get_current_task_id() await api_client_manager.prewarm_assistant_memory_context( @@ -489,6 +509,8 @@ async def submit_outline_update(self) -> Optional[CompilerSubmission]: # Check if update needed needs_update = data.get("needs_update", False) + if type(needs_update) is not bool: + raise ValueError("outline_update needs_update must be a boolean") if not needs_update: # Notify task completed even when no update needed @@ -496,19 +518,37 @@ async def submit_outline_update(self) -> Optional[CompilerSubmission]: self.task_tracking_callback("completed", task_id) logger.info("Outline update not needed") return None - - # Create submission - # Content is already properly decoded by json.loads() - no additional processing needed - content = data.get("content", "") + + operation = data.get("operation", "") + if operation != "insert_after": + raise ValueError( + "outline_update requires operation='insert_after' when needs_update=true" + ) + + old_string = _normalize_string_field(data.get("old_string", "")) + new_string = _normalize_string_field(data.get("new_string", "")) + reasoning = data.get("reasoning", "") + if not old_string.strip(): + raise ValueError( + "outline_update requires non-empty old_string when needs_update=true" + ) + if not new_string.strip(): + raise ValueError( + "outline_update requires non-empty new_string when needs_update=true" + ) + if not isinstance(reasoning, str) or not reasoning.strip(): + raise ValueError( + "outline_update requires non-empty reasoning when needs_update=true" + ) submission = CompilerSubmission( submission_id=str(uuid.uuid4()), mode="outline_update", - content=content, - operation=data.get("operation", "replace"), - old_string=_normalize_string_field(data.get("old_string", "")), - new_string=_normalize_string_field(data.get("new_string", "")), - reasoning=data.get("reasoning", ""), + content=new_string, + operation=operation, + old_string=old_string, + new_string=new_string, + reasoning=reasoning, metadata={} ) @@ -666,6 +706,12 @@ async def submit_construction( rejection_feedback=rejection_feedback ) logger.info(f"Prompt built: {len(prompt)} chars") + from backend.shared.solution_path.integration import with_budgeted_solver_plan + prompt = with_budgeted_solver_plan( + prompt, + getattr(self, "solution_path_manager", None), + max_allowed_tokens, + ) task_id = self.get_current_task_id() await api_client_manager.prewarm_assistant_memory_context( @@ -707,6 +753,8 @@ async def submit_construction( ) except RetryableProviderError: raise + except WolframFinalizationError: + raise except Exception as exc: # Any tool-loop failure falls back to the plain single-shot # path so construction still makes forward progress. @@ -760,9 +808,13 @@ async def submit_construction( # Check if construction needed needs_construction = data.get("needs_construction", True) # Default True for backward compat + if type(needs_construction) is not bool: + raise ValueError("construction needs_construction must be a boolean") # Extract section_complete flag (new phase-based system) section_complete = data.get("section_complete", False) + if type(section_complete) is not bool: + raise ValueError("construction section_complete must be a boolean") if not needs_construction: logger.info(f"Construction not needed - section_complete={section_complete}") @@ -794,24 +846,40 @@ async def submit_construction( self.task_tracking_callback("completed", task_id) return None - # Validate content not empty when needs_construction=True - # The actual content is in "new_string" field, NOT "content" + # Validate exact-edit fields according to the requested operation. + operation = data.get("operation", "full_content") + old_string_content = _normalize_string_field(data.get("old_string", "")) new_string_content = _normalize_string_field(data.get("new_string", "")) - if not new_string_content or not new_string_content.strip(): - logger.warning(f"Construction marked as needed but new_string is empty. Data keys: {list(data.keys())}") + if operation == "delete": + fields_valid = bool(old_string_content.strip()) and not new_string_content.strip() + elif operation in {"replace", "insert_after", "full_content"}: + fields_valid = bool(new_string_content.strip()) + else: + fields_valid = False + + if not fields_valid: + logger.warning( + "Construction has invalid exact-edit fields for operation=%r. Data keys: %s", + operation, + list(data.keys()), + ) # Notify task completed (failed but still completed) if self.task_tracking_callback: self.task_tracking_callback("completed", task_id) return None + + submission_content = ( + old_string_content if operation == "delete" else new_string_content + ) # Create submission with section_complete flag submission = CompilerSubmission( submission_id=str(uuid.uuid4()), mode="construction", - content=new_string_content, # Use new_string as the content - operation=data.get("operation", "full_content"), - old_string=_normalize_string_field(data.get("old_string", "")), - new_string=new_string_content, # Already normalized above + content=submission_content, + operation=operation, + old_string=old_string_content, + new_string=new_string_content, reasoning=data.get("reasoning", ""), section_complete=section_complete, metadata={ @@ -898,6 +966,14 @@ async def submit_review(self, review_focus: str = "general") -> Optional[Compile review_focus=review_focus ) logger.info(f"Prompt built: {len(prompt)} chars") + from backend.shared.solution_path.integration import with_budgeted_solver_plan + prompt = with_budgeted_solver_plan( + prompt, + getattr(self, "solution_path_manager", None), + rag_config.get_available_input_tokens( + self.context_window, self.max_output_tokens + ), + ) task_id = self.get_current_task_id() await api_client_manager.prewarm_assistant_memory_context( @@ -1215,12 +1291,10 @@ async def _generate_completion_with_wolfram_tool( ), }) - # Loop cap reached without a clean finalization - surface whatever - # text the last assistant turn produced, or empty string. - for turn in reversed(messages): - if turn.get("role") == "assistant" and turn.get("content"): - return str(turn["content"]), wolfram_calls, turn - return "", wolfram_calls, {} + raise WolframFinalizationError( + "The writer continued requesting tools and did not produce final JSON " + "after the Wolfram Alpha call budget was exhausted." + ) async def _broadcast_wolfram_event( self, @@ -1372,7 +1446,7 @@ def _build_retry_prompt(self, mode: str, error: str) -> str: "outline_update": ( '{\n' ' "needs_update": true,\n' - ' "operation": "insert_after | replace",\n' + ' "operation": "insert_after",\n' ' "old_string": "exact text from outline (escape backslashes)",\n' ' "new_string": "new sections (LaTeX allowed, escape backslashes)",\n' ' "reasoning": "explanation"\n' diff --git a/backend/compiler/core/compiler_coordinator.py b/backend/compiler/core/compiler_coordinator.py index 4f690c8..25aad65 100644 --- a/backend/compiler/core/compiler_coordinator.py +++ b/backend/compiler/core/compiler_coordinator.py @@ -7,7 +7,7 @@ import re import uuid from pathlib import Path -from typing import Optional, Dict, Callable, List, Tuple +from typing import Any, Optional, Dict, Callable, List, Tuple from datetime import datetime from backend.shared.config import system_config, rag_config @@ -16,7 +16,7 @@ from backend.shared.openrouter_client import FreeModelExhaustedError, OpenRouterInvalidResponseError from backend.shared.brainstorm_proof_gate import BRAINSTORM_LEAN_PROOF_MARKER from backend.shared.free_model_manager import free_model_manager -from backend.shared.json_parser import parse_json +from backend.shared.json_parser import parse_json, sanitize_model_output_for_retry_context from backend.shared.model_error_utils import ( is_non_retryable_model_error, is_transient_model_call_error, @@ -27,6 +27,7 @@ CONTEXT_OVERFLOW_RESOLUTION, CONTEXT_OVERFLOW_STOP_MESSAGE, CONTEXT_OVERFLOW_STOP_REASON, + context_overflow_model_payload, ) from backend.compiler.agents.writer_submitter import WritingSubmitter from backend.compiler.agents.high_param_submitter import HighParamSubmitter @@ -141,12 +142,15 @@ def __init__(self): self.writer_submitter: Optional[WritingSubmitter] = None self.high_param_submitter: Optional[HighParamSubmitter] = None self.validator: Optional[CompilerValidator] = None + self.solution_path_manager = None self.is_running = False self.current_mode = "idle" self.outline_accepted = False self.fatal_error_type: Optional[str] = None self.fatal_error_message: str = "" + self.fatal_error_payload: Optional[Dict[str, Any]] = None + self.top_level_terminal_callback: Optional[Callable[[], Any]] = None # Stats self.total_submissions = 0 @@ -204,6 +208,14 @@ def __init__(self): self.completed_task_ids: set = set() self.current_task_sequence: int = 0 self.current_task_id: Optional[str] = None # Currently executing task + + def _notify_top_level_terminal(self) -> None: + """Notify a route-owned top-level lifecycle without affecting child compilers.""" + if self.top_level_terminal_callback: + try: + self.top_level_terminal_callback() + except Exception: + logger.exception("Compiler top-level terminal callback failed") async def initialize( self, @@ -349,7 +361,7 @@ async def initialize( # from autonomous brainstorm content that may have been loaded in a prior session from backend.aggregator.core.rag_manager import rag_manager logger.info("Clearing RAG for fresh Part 2 compiler session...") - await asyncio.to_thread(rag_manager.clear_all_documents) + await rag_manager.clear_all_documents_async() logger.info("RAG cleared successfully for Part 2 compiler") # Now load the Part 1 aggregator database into clean RAG @@ -388,6 +400,7 @@ async def initialize( websocket_broadcaster=self.websocket_broadcaster, proof_database_store=proof_database if self.autonomous_mode else None, ) + self.writer_submitter.solution_path_manager = self.solution_path_manager await self.writer_submitter.initialize() # Set up task tracking callback for workflow panel integration self.writer_submitter.set_task_tracking_callback(self._handle_task_event) @@ -415,6 +428,7 @@ async def initialize( validator_max_tokens=self.validator_max_tokens, proof_database_store=proof_database if self.autonomous_mode else manual_proof_database, ) + self.high_param_submitter.solution_path_manager = self.solution_path_manager self.high_param_submitter.set_rigor_proof_source( self._current_paper_id or "", self._current_rigor_proof_source_title or compiler_prompt, @@ -474,6 +488,7 @@ async def initialize( compiler_prompt, websocket_broadcaster=self.websocket_broadcaster, proof_database_store=proof_database if self.autonomous_mode else None, + solution_path_manager=self.solution_path_manager, ) await self.validator.initialize() # Set up task tracking callback for workflow panel integration @@ -733,26 +748,29 @@ async def start(self) -> None: self.is_running = True self.fatal_error_type = None self.fatal_error_message = "" + self.fatal_error_payload = None logger.info("Starting compiler...") - # Reset free model manager state for fresh start - free_model_manager.reset() - - # Refresh workflow predictions at start - await self.refresh_workflow_predictions() - - # Start main workflow loop - self._main_task = asyncio.create_task(self._main_workflow()) - - # Manual Part 2 can watch Part 1 for incremental context. Autonomous/Tier 3 - # paper writing owns its brainstorm context and must not spawn child monitors. - if not self.autonomous_mode: - self._aggregator_monitor_task = asyncio.create_task(self._monitor_aggregator_for_rerag()) - else: + try: + free_model_manager.reset() + await self.refresh_workflow_predictions() + self._main_task = asyncio.create_task(self._main_workflow()) + + if not self.autonomous_mode: + self._aggregator_monitor_task = asyncio.create_task(self._monitor_aggregator_for_rerag()) + else: + self._aggregator_monitor_task = None + + await self._broadcast("compiler_started", {"message": "Compiler started"}) + logger.info("Compiler started successfully") + except BaseException: + self.is_running = False + for task in (self._main_task, self._aggregator_monitor_task): + if task and not task.done(): + await _cancel_and_drain_task(task) + self._main_task = None self._aggregator_monitor_task = None - - await self._broadcast("compiler_started", {"message": "Compiler started"}) - logger.info("Compiler started successfully") + raise async def _load_rigor_source_material_context(self) -> tuple[str, str]: """Load direct brainstorm/aggregator context for paper-writing proof mode.""" @@ -784,7 +802,12 @@ async def _load_rigor_source_material_context(self) -> tuple[str, str]: async def stop(self) -> None: """Stop the compiler system.""" - if not self.is_running: + tasks_alive = any( + task is not None and not task.done() + for task in (self._main_task, self._aggregator_monitor_task) + ) + if not self.is_running and not tasks_alive: + self._notify_top_level_terminal() return self.is_running = False @@ -798,26 +821,49 @@ async def stop(self) -> None: if self._main_task: await _cancel_and_drain_task(self._main_task) + self._main_task = None if self._aggregator_monitor_task: await _cancel_and_drain_task(self._aggregator_monitor_task) + self._aggregator_monitor_task = None await self._broadcast("compiler_stopped", {"message": "Compiler stopped"}) logger.info("Compiler stopped") + self._notify_top_level_terminal() async def _handle_context_overflow(self, error: BaseException, *, role_id: str, mode: Optional[str] = None) -> None: """Stop the compiler when mandatory direct context cannot fit.""" self.fatal_error_type = CONTEXT_OVERFLOW_STOP_REASON self.fatal_error_message = str(error) self.is_running = False + config_role_id = { + "compiler_rigor": "compiler_high_param", + }.get(role_id, role_id) + feedback = getattr(error, "feedback", None) + feedback_route_payload = { + key: value + for key, value in { + "configured_model": getattr(feedback, "configured_model", None), + "configured_provider": getattr(feedback, "configured_provider", None), + "effective_model": getattr(feedback, "effective_model", None), + "effective_provider": getattr(feedback, "effective_provider", None), + }.items() + if value + } payload = { + "workflow_mode": "autonomous" if self.autonomous_mode else "compiler", "role_id": role_id, + **context_overflow_model_payload( + api_client_manager.get_role_config(config_role_id) + ), + **feedback_route_payload, "mode": mode or self.current_mode, "reason": CONTEXT_OVERFLOW_STOP_REASON, "message": CONTEXT_OVERFLOW_STOP_MESSAGE, "error_detail": str(error), "resolution": CONTEXT_OVERFLOW_RESOLUTION, } + self.fatal_error_payload = dict(payload) logger.error("Compiler context overflow in %s: %s", role_id, error) await self._broadcast("context_overflow_error", payload) @@ -913,6 +959,9 @@ async def _main_workflow(self) -> None: "mode": self.current_mode, "total_submissions": self.total_submissions }) + finally: + if not self.is_running: + self._notify_top_level_terminal() def _pre_validate_outline_structure(self, content: str) -> Optional[str]: """ @@ -922,10 +971,36 @@ def _pre_validate_outline_structure(self, content: str) -> Optional[str]: This provides IMMEDIATE, CONSISTENT feedback on structural issues without relying on LLM validator interpretation. """ - # Abstract is OPTIONAL - if included, it must be properly formatted - # Valid formats: "Abstract", "I. Abstract", "0. Abstract" (case-insensitive) - # If Abstract is not present, that's also fine - outline can start with Introduction - # We don't enforce Abstract presence here - let validator handle acceptance logic + # Abstract is OPTIONAL. If included, it must be the unnumbered first + # section. Introduction, at least one body section, and Conclusion are + # required in that order. + heading_pattern = re.compile( + r"^\s*(?P(?:[IVXLCDM]+|\d+)\.\s*)?" + r"(?PAbstract|Introduction|Conclusion)\s*$", + re.MULTILINE | re.IGNORECASE, + ) + headings = list(heading_pattern.finditer(content)) + abstract_headings = [ + match for match in headings + if match.group("title").lower() == "abstract" + ] + if abstract_headings: + abstract = abstract_headings[0] + if abstract.group("number"): + return ( + "PRE-VALIDATION FAILED: INCORRECT_SECTION_NAME - Abstract\n\n" + "If included, Abstract must use the unnumbered heading 'Abstract'. " + "Numbered forms such as 'I. Abstract' and '0. Abstract' are invalid." + ) + first_nonempty_line = next( + (line.strip() for line in content.splitlines() if line.strip()), + "", + ) + if first_nonempty_line.lower() != "abstract": + return ( + "PRE-VALIDATION FAILED: INCORRECT_SECTION_ORDER - Abstract\n\n" + "If included, the unnumbered Abstract must be the first section." + ) # Check for "Introduction" header intro_pattern = r'^\s*(?:I\.\s*)?Introduction\s*$' @@ -960,6 +1035,34 @@ def _pre_validate_outline_structure(self, content: str) -> Optional[str]: ❌ Summary ❌ Final Remarks ❌ Closing""" + + intro_match = re.search( + intro_pattern, content, re.MULTILINE | re.IGNORECASE + ) + conclusion_match = re.search( + concl_pattern, content, re.MULTILINE | re.IGNORECASE + ) + if intro_match and conclusion_match and intro_match.start() >= conclusion_match.start(): + return ( + "PRE-VALIDATION FAILED: INCORRECT_SECTION_ORDER\n\n" + "Introduction must appear before the body sections and Conclusion." + ) + body_between = ( + content[intro_match.end():conclusion_match.start()].strip() + if intro_match and conclusion_match + else "" + ) + body_heading_pattern = r"^\s*(?:[IVXLCDM]+|\d+)\.\s+\S.*$" + if not re.search( + body_heading_pattern, + body_between, + re.MULTILINE | re.IGNORECASE, + ): + return ( + "PRE-VALIDATION FAILED: MISSING_REQUIRED_SECTION - Body\n\n" + "Your outline must include at least one numbered body section " + "between Introduction and Conclusion." + ) return None # All critical checks passed @@ -2552,7 +2655,32 @@ async def _place_or_appendix_fallback(self, lean_result) -> bool: placement_outcome="inline", ) try: - await paper_memory.append_to_theorems_appendix(appendix_stub) + stub_appended = await paper_memory.append_to_theorems_appendix( + appendix_stub + ) + if not stub_appended: + await paper_memory.ensure_markers_intact() + stub_appended = await paper_memory.append_to_theorems_appendix( + appendix_stub + ) + if not stub_appended: + logger.error( + "Inline theorem persisted, but appendix stub could not " + "be written after marker repair" + ) + await self._broadcast( + "compiler_rejection", + { + "mode": "rigor", + "submission_id": submission.submission_id, + "reasoning": ( + "The theorem was placed inline, but its Lean " + "appendix entry could not be persisted." + ), + "placement_outcome": "inline_appendix_persistence_failed", + "lean_proof_id": lean_result.proof_id, + }, + ) except Exception as exc: logger.warning( "Inline-placed theorem appendix stub append failed (non-fatal): %s", @@ -2618,8 +2746,8 @@ async def _place_or_appendix_fallback(self, lean_result) -> bool: # Appendix storage: either explicitly requested by discovery or used # as fallback when inline placement failed / was impossible. The math - # is already Lean-verified, so the theorem is preserved and counted as - # a rigor_acceptance. + # is already Lean-verified, but paper-placement success is reported only + # after the appendix entry is durably written. appendix_entry = format_theorem_appendix_entry( proof_id=lean_result.proof_id, theorem_statement=lean_result.theorem_statement, @@ -2638,19 +2766,46 @@ async def _place_or_appendix_fallback(self, lean_result) -> bool: await paper_memory.ensure_markers_intact() appended = await paper_memory.append_to_theorems_appendix(appendix_entry) if not appended: - logger.warning("Appendix append still failed after marker repair; preserving proof record without paper appendix entry") + logger.error( + "Appendix append still failed after marker repair; preserving " + "the canonical proof record without claiming paper persistence" + ) + await self._broadcast( + "compiler_rejection", + { + "mode": "rigor", + "submission_id": ( + lean_result.initial_placement_submission.submission_id + if lean_result.initial_placement_submission + else f"rigor_{appendix_outcome}_{lean_result.proof_id}" + ), + "reasoning": ( + "Lean verification succeeded, but the theorem could not " + "be persisted to the paper appendix after marker repair." + ), + "placement_outcome": "appendix_persistence_failed", + "lean_proof_id": lean_result.proof_id, + }, + ) + return False self.rigor_acceptances += 1 + acceptance_submission_id = ( + lean_result.initial_placement_submission.submission_id + if lean_result.initial_placement_submission + else f"rigor_{appendix_outcome}_{lean_result.proof_id}" + ) + await compiler_rejection_log.add_acceptance( + acceptance_submission_id, + "rigor", + appendix_entry[:500], + ) word_count = await paper_memory.get_word_count() await self._broadcast( "compiler_acceptance", { "mode": "rigor", - "submission_id": ( - lean_result.initial_placement_submission.submission_id - if lean_result.initial_placement_submission - else f"rigor_{appendix_outcome}_{lean_result.proof_id}" - ), + "submission_id": acceptance_submission_id, "placement_outcome": appendix_outcome, "lean_proof_id": lean_result.proof_id, "is_novel": lean_result.is_novel, @@ -3310,6 +3465,8 @@ async def _validate_critique(self, submission) -> Optional[ValidationResult]: ] prompt = ''.join(parts) + from backend.shared.solution_path.integration import with_validator_hook + prompt = with_validator_hook(prompt, self.solution_path_manager) # Generate task ID task_id = self.critique_submitter.next_critique_task_id("critique_val") @@ -3328,8 +3485,34 @@ async def _validate_critique(self, submission) -> Optional[ValidationResult]: message = response.get("choices", [{}])[0].get("message", {}) response_text = extract_message_text(message) - # Parse response - data = parse_json(response_text) + # Parse response, with one bounded same-role JSON repair. + try: + data = parse_json(response_text) + except Exception as parse_error: + safe = sanitize_model_output_for_retry_context( + response_text, max_chars=2000 + ) + repair = ( + "Your previous critique validation response was invalid JSON. " + f"Validation error: {parse_error}. Return the exact same primary " + "decision as one corrected JSON object. Retain the one valid " + "optional top-level `solution_path_update` if present; omission " + "remains valid." + ) + retry = await api_client_manager.generate_completion( + task_id=f"{task_id}_retry", + role_id="critique_validator", + model=self.validator_model, + messages=[ + {"role": "user", "content": prompt}, + {"role": "assistant", "content": safe}, + {"role": "user", "content": repair}, + ], + temperature=0.0, + max_tokens=self.validator_max_tokens, + ) + retry_message = retry.get("choices", [{}])[0].get("message", {}) + data = parse_json(extract_message_text(retry_message)) if data is None: logger.error("Failed to parse critique validation response") @@ -3341,6 +3524,25 @@ async def _validate_critique(self, submission) -> Optional[ValidationResult]: if not data: return None data = data[0] + from backend.shared.solution_path.integration import enqueue_optional_update + critique_decision = str(data.get("decision", "reject")).lower() + reasoning = data.get("reasoning") + if critique_decision not in {"accept", "reject"}: + logger.error("Critique validator returned invalid decision") + return None + if not isinstance(reasoning, str) or not reasoning.strip(): + logger.error("Critique validator returned empty reasoning") + return None + await enqueue_optional_update( + data, + self.solution_path_manager, + proposer_role="compiler_validator", + source_task_id=task_id, + source_phase="critique_validation", + source_decision=( + critique_decision + ), + ) # Create ValidationResult result = ValidationResult( @@ -3389,6 +3591,8 @@ async def _perform_critique_cleanup(self) -> None: ] prompt = ''.join(parts) + from backend.shared.solution_path.integration import with_validator_hook + prompt = with_validator_hook(prompt, self.solution_path_manager) # Call validator task_id = self.critique_submitter.next_critique_task_id("critique_cleanup") @@ -3406,15 +3610,58 @@ async def _perform_critique_cleanup(self) -> None: message = response.get("choices", [{}])[0].get("message", {}) response_text = extract_message_text(message) - # Parse response - data = parse_json(response_text) - - if data is None or not data.get("should_remove", False): + # Parse response, with one bounded same-role JSON repair. + try: + data = parse_json(response_text) + except Exception as parse_error: + safe = sanitize_model_output_for_retry_context( + response_text, max_chars=2000 + ) + repair = ( + "Your previous critique cleanup response was invalid JSON. " + f"Validation error: {parse_error}. Return the exact same primary " + "decision as one corrected JSON object. Retain the one valid " + "optional top-level `solution_path_update` if present; omission " + "remains valid." + ) + retry = await api_client_manager.generate_completion( + task_id=f"{task_id}_retry", + role_id="critique_cleanup", + model=self.validator_model, + messages=[ + {"role": "user", "content": prompt}, + {"role": "assistant", "content": safe}, + {"role": "user", "content": repair}, + ], + temperature=0.0, + max_tokens=self.validator_max_tokens, + ) + retry_message = retry.get("choices", [{}])[0].get("message", {}) + data = parse_json(extract_message_text(retry_message)) + if data is None: + logger.info("Critique cleanup: No removal needed") + return + should_remove = data.get("should_remove", False) + critique_number = data.get("submission_number") + if should_remove and not critique_number: + logger.warning( + "Critique cleanup requested removal without submission_number" + ) + return + from backend.shared.solution_path.integration import enqueue_optional_update + await enqueue_optional_update( + data, + self.solution_path_manager, + proposer_role="compiler_validator", + source_task_id=task_id, + source_phase="critique_cleanup_validation", + source_decision="accept" if should_remove else "reject", + ) + if not should_remove: logger.info("Critique cleanup: No removal needed") return # Remove the critique - critique_number = data.get("submission_number") if critique_number: success = await critique_memory.remove_critique(critique_number) if success: diff --git a/backend/compiler/core/compiler_rag_manager.py b/backend/compiler/core/compiler_rag_manager.py index bad1aae..597b686 100644 --- a/backend/compiler/core/compiler_rag_manager.py +++ b/backend/compiler/core/compiler_rag_manager.py @@ -141,38 +141,18 @@ async def load_aggregator_database(self) -> None: logger.info(f"Initial load: Processing aggregator database with {submission_count} submissions") - # ACQUIRE GLOBAL RAG LOCK FOR ENTIRE INITIAL LOAD - # This prevents the aggregator's re-chunking from interrupting between chunk sizes - await rag_operation_lock.acquire("Compiler initial load (all 4 chunk sizes)") - - try: + async with rag_operation_lock.operation("Compiler initial load (all 4 chunk sizes)"): # Load the aggregator database file directly as a user file # This will chunk it at all 4 configs (256/512/768/1024) aggregator_file_path = str(shared_training_memory.file_path) if Path(aggregator_file_path).exists(): - # Import ingestion pipeline to manually control chunking - from backend.aggregator.ingestion.pipeline import ingestion_pipeline - - # Ingest at all 4 chunk sizes - chunks_by_size = await ingestion_pipeline.ingest_file( + await rag_manager.add_document( aggregator_file_path, - rag_config.submitter_chunk_intervals, # All 4 configs + chunk_sizes=rag_config.submitter_chunk_intervals, is_user_file=True, - trusted_roots=[ - system_config.data_dir, - system_config.user_uploads_dir, - ], ) - # Add all chunks while holding the lock - for chunk_size, chunks in chunks_by_size.items(): - await rag_manager._add_chunks(chunks, chunk_size) - - # Track document - rag_manager.document_count += 1 - rag_manager.permanent_documents.add(Path(aggregator_file_path).name) - logger.info(f"Aggregator database loaded as user file: {submission_count} submissions, 4 chunk configs") else: logger.warning(f"Aggregator database file does not exist: {aggregator_file_path}") @@ -181,10 +161,6 @@ async def load_aggregator_database(self) -> None: self.aggregator_submissions_ragged = submission_count self._aggregator_db_loaded = True - finally: - # ALWAYS RELEASE LOCK - rag_operation_lock.release() - except Exception as e: logger.error(f"Failed to load aggregator database: {e}") raise @@ -196,45 +172,41 @@ async def incremental_rerag_aggregator_database(self) -> None: Removes old chunks and re-adds the entire updated file with global lock. """ try: - # ACQUIRE GLOBAL RAG LOCK - await rag_operation_lock.acquire("Compiler re-RAG aggregator DB") - - # Import here to avoid circular dependency - from backend.aggregator.memory.shared_training import shared_training_memory + async with rag_operation_lock.operation("Compiler re-RAG aggregator DB"): + # Import here to avoid circular dependency + from backend.aggregator.memory.shared_training import shared_training_memory - current_count = await shared_training_memory.get_insights_count() + current_count = await shared_training_memory.get_insights_count() - if current_count <= self.aggregator_submissions_ragged: - logger.info("Incremental re-RAG: No new aggregator submissions to process") - return + if current_count <= self.aggregator_submissions_ragged: + logger.info("Incremental re-RAG: No new aggregator submissions to process") + return - new_submissions_count = current_count - self.aggregator_submissions_ragged - logger.info(f"Incremental re-RAG: Processing {new_submissions_count} new aggregator submissions ({current_count} total)") + new_submissions_count = current_count - self.aggregator_submissions_ragged + logger.info(f"Incremental re-RAG: Processing {new_submissions_count} new aggregator submissions ({current_count} total)") # Remove old aggregator database chunks from RAG - aggregator_file_path = str(shared_training_memory.file_path) - await rag_manager.remove_document("rag_shared_training.txt") + aggregator_file_path = str(shared_training_memory.file_path) + await rag_manager.remove_document(Path(aggregator_file_path).name) # Re-add the entire updated file with all 4 chunk configs - if Path(aggregator_file_path).exists(): - await rag_manager.add_document( - aggregator_file_path, - chunk_sizes=rag_config.submitter_chunk_intervals, # All 4 configs - is_user_file=True # Treated as permanent, never evicted - ) - logger.info(f"Incremental re-RAG complete: {new_submissions_count} new submissions added ({current_count} total)") - else: - logger.error(f"Aggregator database file not found during re-RAG: {aggregator_file_path}") + if Path(aggregator_file_path).exists(): + await rag_manager.add_document( + aggregator_file_path, + chunk_sizes=rag_config.submitter_chunk_intervals, + is_user_file=True + ) + logger.info(f"Incremental re-RAG complete: {new_submissions_count} new submissions added ({current_count} total)") + else: + logger.error(f"Aggregator database file not found during re-RAG: {aggregator_file_path}") + return # Update tracking - self.aggregator_submissions_ragged = current_count + self.aggregator_submissions_ragged = current_count except Exception as e: logger.error(f"Failed to incrementally re-RAG aggregator database: {e}") raise - finally: - # ALWAYS RELEASE LOCK - rag_operation_lock.release() async def retrieve_for_mode( self, diff --git a/backend/compiler/memory/compiler_rejection_log.py b/backend/compiler/memory/compiler_rejection_log.py index 0471290..2a0323a 100644 --- a/backend/compiler/memory/compiler_rejection_log.py +++ b/backend/compiler/memory/compiler_rejection_log.py @@ -155,7 +155,7 @@ async def add_rejection(self, result: CompilerValidationResult, mode: str, submi elif not result.coherence_check: what_to_fix = "\n\nWHAT TO FIX:\nEnsure the content flows naturally with existing document sections. Check for grammatical errors and maintain holistic coherence." elif not result.rigor_check: - what_to_fix = "\n\nWHAT TO FIX:\nEnsure all mathematical claims are based on established principles. Avoid unfounded claims or logical fallacies." + what_to_fix = "\n\nWHAT TO FIX:\nEnsure all claims meet the correctness standard appropriate to their domain and claim type. Mathematical claims require sound derivation, proof, or explicit assumptions; factual, logical, technical, empirical, and methodological claims require appropriate support and must avoid unfounded claims or logical fallacies." entry_text = f"""======================================== 🚫 REJECTED BECAUSE: {failure_reason} diff --git a/backend/compiler/memory/outline_memory.py b/backend/compiler/memory/outline_memory.py index e0b0432..1619a0e 100644 --- a/backend/compiler/memory/outline_memory.py +++ b/backend/compiler/memory/outline_memory.py @@ -31,15 +31,40 @@ class OutlineMemory: """ def __init__(self): - self.file_path = Path(system_config.compiler_outline_file) self.version = 0 self.rechunk_callback: Optional[Callable] = None self._lock = asyncio.Lock() self._initialized = False + self._root_generation = system_config.runtime_root_generation + self._file_path_override: Optional[Path] = None + + @property + def file_path(self) -> Path: + if self._file_path_override is not None: + return self._file_path_override + return Path(system_config.compiler_outline_file) + + @file_path.setter + def file_path(self, value: str | Path) -> None: + path = Path(value) + default_path = Path(system_config.compiler_outline_file) + self._file_path_override = ( + None + if path.resolve(strict=False) == default_path.resolve(strict=False) + else path + ) + + def _refresh_root(self) -> None: + if self._root_generation != system_config.runtime_root_generation: + self.version = 0 + self._initialized = False + self._file_path_override = None + self._root_generation = system_config.runtime_root_generation async def initialize(self) -> None: """Initialize outline memory.""" async with self._lock: + self._refresh_root() if self._initialized: return diff --git a/backend/compiler/memory/paper_memory.py b/backend/compiler/memory/paper_memory.py index 09dbd12..9318c8f 100644 --- a/backend/compiler/memory/paper_memory.py +++ b/backend/compiler/memory/paper_memory.py @@ -43,15 +43,40 @@ class PaperMemory: """ def __init__(self): - self.file_path = Path(system_config.compiler_paper_file) self.version = 0 self.rechunk_callback: Optional[Callable] = None self._lock = asyncio.Lock() self._initialized = False + self._root_generation = system_config.runtime_root_generation + self._file_path_override: Optional[Path] = None + + @property + def file_path(self) -> Path: + if self._file_path_override is not None: + return self._file_path_override + return Path(system_config.compiler_paper_file) + + @file_path.setter + def file_path(self, value: str | Path) -> None: + path = Path(value) + default_path = Path(system_config.compiler_paper_file) + self._file_path_override = ( + None + if path.resolve(strict=False) == default_path.resolve(strict=False) + else path + ) + + def _refresh_root(self) -> None: + if self._root_generation != system_config.runtime_root_generation: + self.version = 0 + self._initialized = False + self._file_path_override = None + self._root_generation = system_config.runtime_root_generation async def initialize(self) -> None: """Initialize paper memory.""" async with self._lock: + self._refresh_root() if self._initialized: return @@ -350,9 +375,9 @@ async def get_previous_versions(self) -> list: """ return [] - def _extract_body_and_appendix(self, paper: str) -> tuple[str, str]: + def _extract_document_regions(self, paper: str) -> tuple[str, str, str]: """ - Split existing paper into (body_lines_text, appendix_body_text). + Split a paper into main body, appendix body, and post-appendix content. The "body" is all non-marker content that sits OUTSIDE the THEOREMS_APPENDIX_START / THEOREMS_APPENDIX_END brackets. The @@ -367,12 +392,14 @@ def _extract_body_and_appendix(self, paper: str) -> tuple[str, str]: paper: Raw paper content Returns: - (body_text, appendix_body_text). Both stripped. Either may be "". + (body_text, appendix_body_text, post_appendix_text), each stripped. """ lines = paper.split('\n') body_lines: list[str] = [] appendix_lines: list[str] = [] + post_appendix_lines: list[str] = [] in_appendix = False + appendix_finished = False for line in lines: if THEOREMS_APPENDIX_START in line: @@ -380,11 +407,17 @@ def _extract_body_and_appendix(self, paper: str) -> tuple[str, str]: continue if THEOREMS_APPENDIX_END in line: in_appendix = False + appendix_finished = True continue if in_appendix: appendix_lines.append(line) continue + + if appendix_finished: + if PAPER_ANCHOR not in line: + post_appendix_lines.append(line) + continue # Outside the appendix: skip structural markers if (ABSTRACT_PLACEHOLDER in line @@ -395,7 +428,11 @@ def _extract_body_and_appendix(self, paper: str) -> tuple[str, str]: body_lines.append(line) - return '\n'.join(body_lines).strip(), '\n'.join(appendix_lines).strip() + return ( + '\n'.join(body_lines).strip(), + '\n'.join(appendix_lines).strip(), + '\n'.join(post_appendix_lines).strip(), + ) def _build_appendix_block(self, appendix_body: str) -> str: """ @@ -455,7 +492,7 @@ async def append_to_theorems_appendix(self, theorem_entry: str) -> bool: ) return False - body_text, appendix_body = self._extract_body_and_appendix(paper) + _, appendix_body, _ = self._extract_document_regions(paper) proof_id_match = re.search(r"(?m)^\s*Proof ID:\s*(\S+)\s*$", entry) if proof_id_match and re.search( rf"(?m)^\s*Proof ID:\s*{re.escape(proof_id_match.group(1))}\s*$", @@ -523,10 +560,16 @@ def _remove_self_review_section(self, content: str) -> str: if start > 0 and content[start] == "\n": start += 1 - anchor_match = re.search(re.escape(PAPER_ANCHOR), content[match.end():]) - end = len(content) - if anchor_match: - end = match.end() + anchor_match.start() + tail = content[match.end():] + boundaries = [ + boundary.start() + for pattern in ( + re.escape(THEOREMS_APPENDIX_START), + re.escape(PAPER_ANCHOR), + ) + if (boundary := re.search(pattern, tail)) + ] + end = match.end() + min(boundaries) if boundaries else len(content) return (content[:start].rstrip() + "\n\n" + content[end:].lstrip()).strip() @@ -683,7 +726,9 @@ def has_real_section_content(section_pattern: str, paper_text: str) -> bool: # Need to add missing placeholders. Preserve any existing appendix # body content while we rebuild the skeleton. - body_content, appendix_body = self._extract_body_and_appendix(paper) + body_content, appendix_body, post_appendix_content = ( + self._extract_document_regions(paper) + ) if not body_content: logger.warning("No body content found - cannot add placeholders to empty paper") @@ -717,6 +762,10 @@ def has_real_section_content(section_pattern: str, paper_text: str) -> bool: # insert the empty placeholder line inside the bracket pair) new_paper_parts.append(self._build_appendix_block(appendix_body)) new_paper_parts.append("") + + if post_appendix_content: + new_paper_parts.append(post_appendix_content) + new_paper_parts.append("") # Anchor at end new_paper_parts.append(PAPER_ANCHOR) @@ -820,7 +869,9 @@ def has_real_section_content(section_pattern: str, paper_text: str) -> bool: return False # Need to repair - preserve body + appendix body across the rebuild - body_content, appendix_body = self._extract_body_and_appendix(paper) + body_content, appendix_body, post_appendix_content = ( + self._extract_document_regions(paper) + ) if not body_content: # Empty paper - just ensure anchor exists @@ -856,6 +907,10 @@ def has_real_section_content(section_pattern: str, paper_text: str) -> bool: # Theorems Appendix (preserve existing entries, default otherwise) new_paper_parts.append(self._build_appendix_block(appendix_body)) new_paper_parts.append("") + + if post_appendix_content: + new_paper_parts.append(post_appendix_content) + new_paper_parts.append("") # Anchor at end new_paper_parts.append(PAPER_ANCHOR) diff --git a/backend/compiler/prompts/construction_prompts.py b/backend/compiler/prompts/construction_prompts.py index 26e31b4..01fbb65 100644 --- a/backend/compiler/prompts/construction_prompts.py +++ b/backend/compiler/prompts/construction_prompts.py @@ -1,5 +1,5 @@ """ -Construction prompts for mathematical document building. +Construction prompts for rigorous solution-document building. Implements phase-based construction with explicit section_complete feedback. PHASE ORDER (enforced): @@ -15,14 +15,17 @@ CONSTRUCTION_EMPIRICAL_PROVENANCE_RULES = """EMPIRICAL PROVENANCE RULES: -- Classify substantive claims as one of: theoretical claim, literature claim, empirical claim, or artifact claim. -- Theoretical claims must be supported by sound derivation, proof, or explicit assumptions inside the paper. +- Apply claim-type and domain-appropriate rigor to every substantive claim. +- Mathematical claims require sound derivation, proof, or explicit assumptions inside the paper. +- Engineering and software proposals require concrete mechanisms, constraints, interfaces, failure modes, feasibility reasoning, and verification plans appropriate to the claim. +- Strategic or causal claims require valid inference, explicit assumptions, and realistic limitations. - Literature claims must include explicit in-text citations identifying the external source. - Empirical claims include benchmark results, latency, throughput, speedups, accuracy, perplexity, ablation outcomes, hardware utilization, and measured implementation metrics. - Artifact claims include statements about code, kernels, experiments, logs, reproductions, or accompanying implementations. - Empirical or artifact claims may be stated as facts ONLY when backed by an explicit external citation or a provided artifact in context. - If that support is missing, rewrite the material as a hypothesis, expected benefit, design target, proposed experiment, validation plan, limitation, or future work. -- NEVER invent citations, experiments, benchmark numbers, hardware measurements, datasets, or code artifacts.""" +- NEVER invent citations, experiments, benchmark numbers, hardware measurements, datasets, or code artifacts. +- Novelty never overrides correctness, provenance, safety, or honesty.""" def get_wolfram_tool_guidance() -> str: @@ -65,24 +68,13 @@ def get_wolfram_tool_guidance() -> str: When you use the tool, incorporate only relevant verified results into your final JSON `new_string` and explain in `reasoning` how the Wolfram result informed the content. The system records the full audit trail separately.""" -CONSTRUCTION_EMPIRICAL_PROVENANCE_RULES = """EMPIRICAL PROVENANCE RULES: -- Classify substantive claims as one of: theoretical claim, literature claim, empirical claim, or artifact claim. -- Theoretical claims must be supported by sound derivation, proof, or explicit assumptions inside the paper. -- Literature claims must include explicit in-text citations identifying the external source. -- Empirical claims include benchmark results, latency, throughput, speedups, accuracy, perplexity, ablation outcomes, hardware utilization, and measured implementation metrics. -- Artifact claims include statements about code, kernels, experiments, logs, reproductions, or accompanying implementations. -- Empirical or artifact claims may be stated as facts ONLY when backed by an explicit external citation or a provided artifact in context. -- If that support is missing, rewrite the material as a hypothesis, expected benefit, design target, proposed experiment, validation plan, limitation, or future work. -- NEVER invent citations, experiments, benchmark numbers, hardware measurements, datasets, or code artifacts.""" - - # ============================================================================= # PHASE-SPECIFIC CONSTRUCTION PROMPTS # ============================================================================= def get_body_construction_system_prompt() -> str: """Get system prompt for BODY section construction phase.""" - return """You are constructing the BODY SECTIONS of a mathematical document. Your ONLY task in this phase is to write main content sections. + return """You are constructing the BODY SECTIONS of a rigorous, solution-oriented research paper or report. Your ONLY task in this phase is to write main solution content. ⚠️ CRITICAL - INTERNAL CONTENT WARNING ⚠️ @@ -97,7 +89,7 @@ def get_body_construction_system_prompt() -> str: """ + CONSTRUCTION_EMPIRICAL_PROVENANCE_RULES + """ - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been proven correct. Aggressively pursue the strongest credible and genuinely novel solution to the user's exact objective. Choose the solution form and verification standard that fit the problem. Mathematical reasoning, theorems, derivations, proofs, and LaTeX remain first-class and preferred whenever relevant, but non-mathematical work must not be forced into mathematical form. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -105,17 +97,17 @@ def get_body_construction_system_prompt() -> str: IMPORTANT - WHY WE WRITE PAPERS OUT OF ORDER: This paper is constructed OUT OF ORDER intentionally. The writing sequence is: -1. BODY SECTIONS FIRST - establishes the actual mathematical content with full flexibility -2. CONCLUSION second - summarizes what was actually proven in the body +1. BODY SECTIONS FIRST - establishes the actual solution content with full flexibility +2. CONCLUSION second - summarizes what was actually established in the body 3. INTRODUCTION third - describes what the paper actually contains (written after we know the content) 4. ABSTRACT last - summarizes the complete paper -WHY BODY FIRST? If we wrote the introduction or abstract first, it would lock in what the body must contain before we've written it. By writing body sections first, the mathematical content can develop naturally and organically without being constrained by promises made in a pre-written introduction. +WHY BODY FIRST? If we wrote the introduction or abstract first, it would lock in what the body must contain before we've written it. By writing body sections first, the solution can develop naturally without being constrained by promises made in a pre-written introduction. SOURCE USAGE PRINCIPLE: - Treat the brainstorm/aggregator database as optional high-value source material and exploration history, not a mandatory checklist - Use it when it helps you achieve the strongest rigorous paper toward the user's prompt -- You may synthesize beyond brainstorm/database material using sound mathematical reasoning +- You may synthesize beyond brainstorm/database material using sound domain-appropriate reasoning - Do NOT force coverage of every source entry - Do NOT ignore clearly crucial source material for the scope you are writing @@ -156,7 +148,7 @@ def get_body_construction_system_prompt() -> str: WHY THEY EXIST: They prevent AI confusion about whether sections like Introduction are "already written" when they aren't. PHASE: BODY SECTIONS -You are writing the main content of the paper - definitions, theorems, proofs, results, discussions. +You are writing the paper's main solution content. Use the modality best suited to the objective: for example mathematical definitions/theorems/proofs; algorithmic constraints/architecture/correctness/interfaces/complexity/evaluation; engineering requirements/mechanisms/tradeoffs/failure modes/verification; empirical hypotheses/protocols/falsifiable predictions/analysis plans/limitations; or strategic objectives/causal models/interventions/risks/evaluation. YOUR TASK: 1. Review the outline to identify which body sections need to be written @@ -167,11 +159,10 @@ def get_body_construction_system_prompt() -> str: PROGRESSIVE SYSTEM: You will be called repeatedly — once per body section. Focus on writing ONE complete, rigorous section per turn rather than rushing through multiple sections. Write what you can do thoroughly and correctly this turn; you will be called again for the next section. WHAT COUNTS AS BODY SECTIONS: -- Definitions and Preliminaries -- Main Results / Theorems -- Proofs -- Corollaries -- Discussion of results +- Every central solution component promised by the outline +- Definitions, theorems, proofs, derivations, and corollaries when the objective is mathematical +- Algorithms, architectures, mechanisms, interfaces, correctness or feasibility arguments, evaluation plans, evidence analysis, failure modes, risks, and limitations when appropriate +- Discussion that directly supports the answer-bearing route - Any numbered sections from the outline EXCEPT Introduction, Conclusion, and Abstract DO NOT WRITE IN THIS PHASE: @@ -181,13 +172,13 @@ def get_body_construction_system_prompt() -> str: COMPLETION CRITERIA - Set section_complete=true when: ✓ ALL body sections listed in the outline have been written -✓ All theorems, proofs, and results from the outline are present -✓ The main mathematical content is complete +✓ Every central solution component promised by the outline is present +✓ Any relevant theorems, proofs, derivations, mechanisms, evidence, interfaces, evaluation plans, and failure analyses are complete enough for the claimed result Set section_complete=false if: ✗ There are still body sections in the outline that haven't been written -✗ Important theorems or proofs are missing -✗ The main content is incomplete +✗ Any important promised solution component is missing, including a theorem/proof when the mathematical route requires it +✗ The main answer-bearing content is incomplete CRITICAL REQUIREMENTS: - Follow the outline structure for body sections @@ -196,8 +187,8 @@ def get_body_construction_system_prompt() -> str: - Prioritize the strongest direct rigorous route to answering the user's prompt - Do not repeat content already in the document - Check for existing section headers before creating new ones -- Write clear, rigorous mathematical exposition -- ALL content must be rooted in sound mathematical reasoning +- Write clear, rigorous, independently checkable, domain-appropriate exposition +- Mathematical claims require sound derivation, proof, or explicit assumptions; other claims require the verification standard appropriate to their type - Unsupported empirical or artifact claims must be rewritten as hypotheses, validation plans, limitations, or future work instead of being asserted as completed results EXACT STRING MATCHING FOR EDITS: @@ -244,11 +235,11 @@ def get_body_construction_system_prompt() -> str: 🚨 CRITICAL CONTENT REQUIREMENT 🚨 -If you set needs_construction=true, you MUST provide actual content in new_string. -- needs_construction=true + new_string="" is INVALID and will be rejected -- needs_construction=true means you ARE writing content - so PROVIDE it +If you set needs_construction=true, you MUST provide actual content in new_string unless operation="delete". +- needs_construction=true + new_string="" is INVALID except for delete, where old_string identifies the content to remove +- needs_construction=true means you ARE editing content - provide new_string for full_content, replace, and insert_after - Only set needs_construction=false if NO body sections remain to write -- The "new_string" field must NEVER be empty when needs_construction=true +- The "new_string" field must be non-empty when needs_construction=true except for delete WRONG (will be rejected): { @@ -270,7 +261,7 @@ def get_body_construction_system_prompt() -> str: "section_complete": true or false, "operation": "full_content | replace | insert_after | delete", "old_string": "exact text from document to find (empty for full_content)", - "new_string": "Your complete section text (MUST NOT be empty if needs_construction=true)", + "new_string": "Your complete section text (non-empty if needs_construction=true, except empty for delete)", "reasoning": "Why construction is/isn't needed AND whether body phase is complete" } """ @@ -278,7 +269,7 @@ def get_body_construction_system_prompt() -> str: def get_conclusion_construction_system_prompt() -> str: """Get system prompt for CONCLUSION section construction phase.""" - return """You are constructing the CONCLUSION section of a mathematical document. Your ONLY task in this phase is to write the conclusion. + return """You are constructing the CONCLUSION section of a rigorous, solution-oriented research paper or report. Your ONLY task in this phase is to write the conclusion. ⚠️ CRITICAL - INTERNAL CONTENT WARNING ⚠️ @@ -293,7 +284,7 @@ def get_conclusion_construction_system_prompt() -> str: """ + CONSTRUCTION_EMPIRICAL_PROVENANCE_RULES + """ - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been proven correct. Use claim-type and domain-appropriate rigor to summarize only what the body actually establishes. Mathematical results and proofs remain fully supported when relevant. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -301,12 +292,12 @@ def get_conclusion_construction_system_prompt() -> str: IMPORTANT - WHY WE WRITE PAPERS OUT OF ORDER: This paper is constructed OUT OF ORDER intentionally. The writing sequence is: -1. BODY SECTIONS first - establishes the actual mathematical content (DONE) -2. CONCLUSION second - YOU ARE HERE - summarize what was actually proven +1. BODY SECTIONS first - establishes the actual solution content (DONE) +2. CONCLUSION second - YOU ARE HERE - summarize what was actually established 3. INTRODUCTION third - will describe what the paper contains (written after we know full content) 4. ABSTRACT last - will summarize the complete paper -WHY CONCLUSION BEFORE INTRODUCTION? Writing the conclusion now, before the introduction, ensures we summarize actual proven results rather than hypothetical content. The introduction will be written next, and it will accurately describe both the body content AND this conclusion. +WHY CONCLUSION BEFORE INTRODUCTION? Writing the conclusion now ensures we summarize actual established results rather than hypothetical content. The introduction will be written next and will accurately describe both the body and this conclusion. CRITICAL - SYSTEM-MANAGED MARKERS (NOT YOUR OUTPUT): @@ -347,7 +338,7 @@ def get_conclusion_construction_system_prompt() -> str: DO NOT RESPOND WITH needs_construction=false. You are in the CONCLUSION PHASE which means: 1. YOU MUST WRITE THE CONCLUSION SECTION NOW 2. SET needs_construction=true -3. PROVIDE THE ACTUAL CONCLUSION TEXT in the "content" field +3. PROVIDE THE ACTUAL CONCLUSION TEXT in the "new_string" field 4. SET section_complete=true (because writing the conclusion completes this phase) WRONG RESPONSE (DO NOT DO THIS): @@ -375,19 +366,19 @@ def get_conclusion_construction_system_prompt() -> str: 2. WRITE the conclusion content that summarizes the main results and contributions 3. SET needs_construction=true (because you ARE constructing content) 4. SET section_complete=true (because writing the conclusion completes this phase) -5. PROVIDE the actual Conclusion text in the "content" field +5. PROVIDE the actual Conclusion text in the "new_string" field DIRECT-ANSWER-FIRST REQUIREMENT: - Make the paper's strongest justified answer, partial answer, impossibility result, or sharp constraint explicit - Do not hide the core answer behind generic summary language WHAT TO INCLUDE IN CONCLUSION: -- Summary of main results and theorems proven +- Summary of the main results, mechanisms, evidence, designs, theorems, or proofs actually established, as applicable - Clear statement of the strongest direct answer the paper has established -- Significance of the mathematical contributions +- Significance of the paper's contributions - Connections between results - Brief mention of limitations or open questions (optional) -- Final remarks on the mathematical significance +- Final remarks on significance and applicability - If empirical validation was not actually supported, state the limitation plainly instead of summarizing unsupported benchmark claims as established fact CRITICAL - SECTION HEADER FORMAT: @@ -412,12 +403,12 @@ def get_conclusion_construction_system_prompt() -> str: CRITICAL REQUIREMENTS: - You MUST set needs_construction=true -- You MUST provide the actual Conclusion text in "content" +- You MUST provide the actual Conclusion text in "new_string" - You MUST set section_complete=true (writing conclusion = completing phase) -- Do NOT add new theorems or proofs - those belong in body sections -- Summarize, don't introduce new material +- Do NOT add any new central result, mechanism, evidence, theorem, or proof - those belong in body sections +- Summarize; do not introduce new material - Maintain coherent narrative flow from body to conclusion -- Write clear, rigorous mathematical exposition +- Write clear, rigorous, domain-appropriate exposition - Do not convert unsupported empirical ideas into factual claims while summarizing EXACT STRING MATCHING FOR EDITS: @@ -440,7 +431,7 @@ def get_conclusion_construction_system_prompt() -> str: def get_introduction_construction_system_prompt() -> str: """Get system prompt for INTRODUCTION section construction phase.""" - return """You are constructing the INTRODUCTION section of a mathematical document. Your ONLY task in this phase is to write the introduction. + return """You are constructing the INTRODUCTION section of a rigorous, solution-oriented research paper or report. Your ONLY task in this phase is to write the introduction. ⚠️ CRITICAL - INTERNAL CONTENT WARNING ⚠️ @@ -455,7 +446,7 @@ def get_introduction_construction_system_prompt() -> str: """ + CONSTRUCTION_EMPIRICAL_PROVENANCE_RULES + """ - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been proven correct. Use claim-type and domain-appropriate rigor, and describe only solution content the completed body and conclusion actually support. Mathematical reasoning, theorems, proofs, and LaTeX remain fully supported when relevant. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -463,8 +454,8 @@ def get_introduction_construction_system_prompt() -> str: IMPORTANT - WHY WE WRITE PAPERS OUT OF ORDER: This paper is constructed OUT OF ORDER intentionally. The writing sequence is: -1. BODY SECTIONS first - established the actual mathematical content (DONE) -2. CONCLUSION second - summarized what was proven (DONE) +1. BODY SECTIONS first - established the actual solution content (DONE) +2. CONCLUSION second - summarized what was established (DONE) 3. INTRODUCTION third - YOU ARE HERE - describe what the paper contains 4. ABSTRACT last - will summarize the complete paper @@ -509,7 +500,7 @@ def get_introduction_construction_system_prompt() -> str: DO NOT RESPOND WITH needs_construction=false. You are in the INTRODUCTION PHASE which means: 1. YOU MUST WRITE THE INTRODUCTION SECTION NOW 2. SET needs_construction=true -3. PROVIDE THE ACTUAL INTRODUCTION TEXT in the "content" field +3. PROVIDE THE ACTUAL INTRODUCTION TEXT in the "new_string" field 4. SET section_complete=true (because writing the introduction completes this phase) WRONG RESPONSE (DO NOT DO THIS): @@ -545,16 +536,16 @@ def get_introduction_construction_system_prompt() -> str: 3. WRITE the introduction that properly introduces and describes the paper's content 4. SET needs_construction=true (because you ARE constructing content) 5. SET section_complete=true (because writing the introduction completes this phase) -6. PROVIDE the actual Introduction text in the "content" field +6. PROVIDE the actual Introduction text in the "new_string" field DIRECT-ANSWER-FIRST REQUIREMENT: - Frame the paper around the main question and the direct answer the body/conclusion establish - Keep preliminaries and motivation in service of that answer rather than drifting into generic survey exposition WHAT TO INCLUDE IN INTRODUCTION: -- Context and motivation for the mathematical problem +- Context and motivation for the exact problem - Brief overview of what the paper covers -- Statement of main results (high-level, not full proofs) +- Statement of main results or solution contributions (high-level; do not duplicate full proofs or implementation detail) - Clear framing of the paper's answer-bearing contribution - Roadmap of the paper structure - Historical context or prior work (if relevant) @@ -562,7 +553,7 @@ def get_introduction_construction_system_prompt() -> str: CRITICAL - SECTION HEADER FORMAT: - Use EXACTLY "I. Introduction" as the section header - The Introduction is ALWAYS Section I (Roman numeral one) -- This follows standard mathematical paper conventions +- This follows the compiler's required paper convention DO NOT WRITE IN THIS PHASE: - Additional body content (that phase is complete) @@ -580,11 +571,11 @@ def get_introduction_construction_system_prompt() -> str: CRITICAL REQUIREMENTS: - You MUST set needs_construction=true -- You MUST provide the actual Introduction text in "content" +- You MUST provide the actual Introduction text in "new_string" - You MUST set section_complete=true (writing introduction = completing phase) - The introduction is the ONLY place where forward-looking language is allowed -- Describe results without full proofs -- Set up the mathematical context +- Describe results without duplicating full proofs or detailed body argument +- Set up the domain-appropriate problem context - Make the reader want to continue reading - Do not promise empirical validation, benchmark numbers, or artifacts unless they are explicitly supported @@ -608,7 +599,7 @@ def get_introduction_construction_system_prompt() -> str: def get_abstract_construction_system_prompt() -> str: """Get system prompt for ABSTRACT section construction phase.""" - return """You are constructing the ABSTRACT of a mathematical document. Your ONLY task in this phase is to write the abstract. + return """You are constructing the ABSTRACT of a rigorous, solution-oriented research paper or report. Your ONLY task in this phase is to write the abstract. ⚠️ CRITICAL - INTERNAL CONTENT WARNING ⚠️ @@ -623,7 +614,7 @@ def get_abstract_construction_system_prompt() -> str: """ + CONSTRUCTION_EMPIRICAL_PROVENANCE_RULES + """ - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been proven correct. Use claim-type and domain-appropriate rigor and summarize only what the completed paper actually supports. Mathematical reasoning, theorems, proofs, and LaTeX remain fully supported when relevant. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -631,8 +622,8 @@ def get_abstract_construction_system_prompt() -> str: IMPORTANT - WHY WE WRITE PAPERS OUT OF ORDER: This paper is constructed OUT OF ORDER intentionally. The writing sequence is: -1. BODY SECTIONS first - established the actual mathematical content (DONE) -2. CONCLUSION second - summarized what was proven (DONE) +1. BODY SECTIONS first - established the actual solution content (DONE) +2. CONCLUSION second - summarized what was established (DONE) 3. INTRODUCTION third - described what the paper contains (DONE) 4. ABSTRACT last - YOU ARE HERE - summarize the complete paper @@ -676,7 +667,7 @@ def get_abstract_construction_system_prompt() -> str: DO NOT RESPOND WITH needs_construction=false. You are in the ABSTRACT PHASE (THE FINAL PHASE) which means: 1. YOU MUST WRITE THE ABSTRACT SECTION NOW 2. SET needs_construction=true -3. PROVIDE THE ACTUAL ABSTRACT TEXT in the "content" field +3. PROVIDE THE ACTUAL ABSTRACT TEXT in the "new_string" field 4. ALWAYS SET section_complete=true (this is the FINAL phase - writing abstract = completing paper) WRONG RESPONSE (DO NOT DO THIS): @@ -712,7 +703,7 @@ def get_abstract_construction_system_prompt() -> str: 3. WRITE a concise abstract that summarizes the entire paper 4. SET needs_construction=true (because you ARE constructing content) 5. ALWAYS SET section_complete=true (THIS IS THE FINAL PHASE - no exceptions) -6. PROVIDE the actual Abstract text in the "content" field +6. PROVIDE the actual Abstract text in the "new_string" field DIRECT-ANSWER-FIRST REQUIREMENT: - State the strongest direct answer, partial answer, impossibility result, or sharp constraint up front when the paper justifies it @@ -749,7 +740,7 @@ def get_abstract_construction_system_prompt() -> str: CRITICAL REQUIREMENTS: - You MUST set needs_construction=true -- You MUST provide the actual Abstract text in "content" +- You MUST provide the actual Abstract text in "new_string" - You MUST ALWAYS set section_complete=true (this is the final phase - no exceptions) - The abstract should stand alone - reader should understand contributions without reading the paper - Be concise but comprehensive @@ -786,7 +777,7 @@ def get_construction_system_prompt() -> str: NOTE: For autonomous mode and proper section ordering, use phase-specific prompts instead. """ - return """You are constructing a mathematical document section by section. Your role is to: + return """You are constructing a rigorous, solution-oriented research paper or report section by section. Your role is to: ⚠️ CRITICAL - INTERNAL CONTENT WARNING ⚠️ @@ -801,7 +792,7 @@ def get_construction_system_prompt() -> str: """ + CONSTRUCTION_EMPIRICAL_PROVENANCE_RULES + """ - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been proven correct. Aggressively pursue the strongest credible and genuinely novel solution to the user's exact objective. Choose the solution form and verification standard that fit the problem. Mathematical reasoning, theorems, derivations, proofs, and LaTeX remain first-class whenever relevant. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -842,10 +833,10 @@ def get_construction_system_prompt() -> str: - Use brainstorm/aggregator content when it helps, but you are not required to cover every source entry - Prioritize the strongest direct rigorous route to answering the user's prompt - Maintain coherent narrative flow -- Write clear, rigorous mathematical exposition +- Write clear, rigorous, independently checkable, domain-appropriate exposition - Do not repeat content already in the document - Check for existing section headers before creating new ones -- ALL content must be rooted in sound mathematical reasoning +- Apply claim-type rigor: mathematical claims require derivation/proof/assumptions; engineering, software, empirical, and strategic claims require their appropriate mechanisms, evidence, constraints, failure modes, and verification - Unsupported empirical or artifact claims must be rewritten conservatively rather than asserted as established fact EXACT STRING MATCHING FOR EDITS: @@ -885,7 +876,7 @@ def get_construction_json_schema() -> str: } FIELD DEFINITIONS: -- needs_construction: Set to false if no more content is needed in the CURRENT PHASE +- needs_construction: Set to false only if no more content is needed in the CURRENT PHASE. If true, new_string contains the actual edit content except for delete, where it is empty. - section_complete: Set to true when the CURRENT PHASE (body/conclusion/intro/abstract) is complete - operation: Type of edit operation: * "full_content": Replace entire document with new_string (for first content) @@ -895,7 +886,7 @@ def get_construction_json_schema() -> str: - old_string: EXACT text from the current document that you want to find. Must be unique. Include enough context (3-5 lines) to ensure uniqueness. Empty for full_content operation. - new_string: The actual text to insert (for insert_after/full_content) or replace with (for replace). - Empty for delete operation. + It MUST be non-empty when needs_construction=true, except for a delete operation; empty when needs_construction=false. - reasoning: Explain your decision and phase status EXACT STRING MATCHING RULES: @@ -931,9 +922,9 @@ def get_construction_json_schema() -> str: "needs_construction": true, "section_complete": false, "operation": "insert_after", - "old_string": "This completes the proof of Theorem 2.1.\\n\\n[HARD CODED PLACEHOLDER FOR THE CONCLUSION", - "new_string": "III. Main Results\\n\\nWe now present the central theorem...", - "reasoning": "Section III Main Results is needed per the outline. Inserting after Section II ends." + "old_string": "The interface contract above fixes the request and response invariants.", + "new_string": "\\n\\nIII. Failure Modes and Verification\\n\\nWe now analyze degraded dependencies, recovery behavior, and the tests needed to verify each invariant.", + "reasoning": "The outline requires failure-mode and verification analysis after the architecture section. The editable prose anchor is exact and contains no protected marker." } Example (Replacing a paragraph): @@ -1098,7 +1089,7 @@ async def build_construction_prompt( parts.append("""OPTIONAL SOURCE MATERIAL POLICY: - The brainstorm database and source evidence below are optional supports, not mandatory checklists. - Use them if they help you achieve the strongest rigorous paper toward the user's prompt. -- You may synthesize beyond them using sound mathematical reasoning. +- You may synthesize beyond them using sound domain-appropriate reasoning; mathematical reasoning and proof remain first-class when relevant. - Do NOT force coverage of every source entry. - Prefer material that strengthens the paper's direct answer over broader auxiliary coverage. """) diff --git a/backend/compiler/prompts/critique_prompts.py b/backend/compiler/prompts/critique_prompts.py index 6679d3e..89c85d9 100644 --- a/backend/compiler/prompts/critique_prompts.py +++ b/backend/compiler/prompts/critique_prompts.py @@ -8,6 +8,8 @@ CRITIQUE_EMPIRICAL_PROVENANCE_RULES = """EMPIRICAL / ARTIFACT CLAIM POLICY: +- Apply claim-type and domain-appropriate rigor. Mathematical claims require sound derivation, proof, or explicit assumptions; engineering/software claims require mechanisms, constraints, interfaces, failure modes, feasibility reasoning, and verification plans; strategic or causal claims require valid inference, explicit assumptions, and realistic limitations. +- Literature claims require identifiable external sources rather than invented citations. - Artifact claims include statements about code, kernels, experiments, logs, reproductions, or accompanying implementations. - Empirical or artifact claims may be accepted as factual ONLY when backed by an explicit external citation or a provided artifact in context. - If such support is absent, they should be criticized, removed, or reframed as hypotheses, validation plans, expected benefits, limitations, or future work. @@ -16,7 +18,7 @@ def get_critique_submitter_system_prompt() -> str: """System prompt for generating self-review critiques of the body section.""" - return """You are a peer reviewer generating constructive self-review notes for a mathematical document's body section. + return """You are a peer reviewer generating constructive self-review notes for a rigorous, solution-oriented research paper or report's body section. IMPORTANT - INTERNAL CONTENT WARNING: @@ -39,11 +41,12 @@ def get_critique_submitter_system_prompt() -> str: SOURCE MATERIAL POLICY: - The aggregator/brainstorm database and reference papers are optional support for critique, not mandatory checklists. - Do NOT critique solely because the body does not explicitly cover some source material. -- Do critique omitted material when the omission creates a genuine gap relative to the current outline, stated paper scope, or mathematical goals. +- Do critique omitted material when the omission creates a genuine gap relative to the current outline, stated paper scope, or direct solution goal. - Focus on whether the paper itself is strong, rigorous, and aligned, not on exhaustively mirroring source inputs. CRITIQUE QUALITY REQUIREMENTS: -- Identify only substantive mathematical, logical, structural, or provenance issues. +- Identify only substantive mathematical, logical, technical, algorithmic, engineering, design, empirical, implementation, structural, provenance, or directness issues, as appropriate to the objective. +- Mathematical errors and proof gaps remain first-class critique targets whenever applicable; non-mathematical work must not be criticized merely for lacking theorem/proof form. - Be specific enough that a reader understands the limitation or concern. - Do not propose direct edits or rewrites. The critique will be appended transparently as self-review. - Do not list every possible issue. You will be called up to 3 total attempts, so focus on one important point per turn. @@ -78,8 +81,8 @@ def get_critique_json_schema() -> str: Example critique: { "critique_needed": true, - "submission": "Section III asserts a convergence claim without establishing the needed uniform bound. This is a substantive limitation because later arguments depend on that convergence statement. The paper should be read with this proof gap in mind unless an independent bound is supplied.", - "reasoning": "This is a mathematical gap that affects the reliability of a downstream claim and is suitable for the self-review section." + "submission": "The architecture assumes idempotent retries but does not specify an idempotency-key lifecycle or a recovery test for partial downstream commits. This leaves a substantive correctness and implementation gap in the proposed failure-handling mechanism.", + "reasoning": "This concrete engineering limitation affects whether the central design works under the failure mode it claims to handle and is suitable for the self-review section." } Example decline: @@ -93,7 +96,7 @@ def get_critique_json_schema() -> str: def get_critique_validator_system_prompt() -> str: """System prompt for validating critique submissions.""" - return """You are a validation agent reviewing peer-review critiques for a mathematical document's self-review section. + return """You are a validation agent reviewing peer-review critiques for a rigorous, solution-oriented research paper or report's self-review section. IMPORTANT - INTERNAL CONTENT WARNING: @@ -116,7 +119,7 @@ def get_critique_validator_system_prompt() -> str: For DECLINE ASSESSMENTS (critique_needed=false): evaluate whether the submitter's assessment that no substantive critique is needed is correct. ACCEPT a critique if it: -1. Identifies a real mathematical error, proof gap, unsupported claim, structural problem, or material limitation. +1. Identifies a real mathematical error, proof gap, logical or technical flaw, design/implementation/empirical/provenance/directness problem, unsupported claim, structural problem, or material limitation, as applicable. 2. Is specific and useful to readers. 3. Is substantive rather than stylistic. 4. Is non-redundant with existing accepted critiques. @@ -130,7 +133,7 @@ def get_critique_validator_system_prompt() -> str: 5. Criticizes selective non-use of optional source material without a real gap in the paper's stated scope. 6. Is trivial or pedantic without meaningful impact. -For declines, ACCEPT only if the body is academically acceptable and any remaining issues are minor. REJECT if a substantive issue was missed. +For declines, ACCEPT only if the body is rigorous and acceptable for its stated scope and objective and any remaining issues are minor. REJECT if a substantive domain-appropriate issue was missed. Output your decision ONLY as JSON in this exact format: { @@ -179,8 +182,6 @@ def build_critique_prompt( "\n---\n", f"USER COMPILER-DIRECTING PROMPT:\n{user_prompt}", "\n---\n", - f"PAPER TITLE:\n{user_prompt}", - "\n---\n", f"CURRENT OUTLINE:\n{current_outline}", "\n---\n", f"CURRENT BODY SECTION (to critique):\n{current_body}", diff --git a/backend/compiler/prompts/outline_prompts.py b/backend/compiler/prompts/outline_prompts.py index ffc4fbd..809533e 100644 --- a/backend/compiler/prompts/outline_prompts.py +++ b/backend/compiler/prompts/outline_prompts.py @@ -1,11 +1,12 @@ """ -Outline prompts for mathematical document structure generation. +Outline prompts for rigorous solution-document structure generation. """ from backend.compiler.memory.compiler_rejection_log import compiler_rejection_log OUTLINE_EMPIRICAL_PROVENANCE_RULES = """EMPIRICAL PROVENANCE RULES FOR OUTLINES: +- Match each claim to its appropriate verification standard. Mathematical claims require sound derivation, proof, or explicit assumptions; engineering/software proposals require mechanisms, constraints, failure modes, feasibility reasoning, and verification plans; strategic or causal claims require valid inference, explicit assumptions, and realistic limitations. - Do NOT turn unsupported benchmark-style claims into committed outline sections. - Numeric empirical claims in headings or subsection titles (speedup, latency, throughput, perplexity, accuracy, hardware measurements, benchmark names, etc.) are forbidden unless explicitly backed by an external citation or provided artifact in context. - If empirical support is missing, describe the material conservatively as a proposed evaluation, validation plan, expected benefit, design target, future-work task, or open question. @@ -15,7 +16,7 @@ def get_outline_create_system_prompt() -> str: """Get system prompt for initial outline creation.""" - return """You are creating the initial outline for a mathematical document. Your role is to: + return """You are creating the initial outline for a rigorous, solution-oriented research paper or report. Your role is to: 1. Review the aggregated database (accepted submissions from the aggregator tool) 2. Review the user's compiler-directing prompt @@ -34,7 +35,7 @@ def get_outline_create_system_prompt() -> str: """ + OUTLINE_EMPIRICAL_PROVENANCE_RULES + """ - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been proven correct. Aggressively pursue the strongest credible and genuinely novel solution to the user's exact objective. Choose the document structure, solution form, and verification standard that fit the problem. Mathematical reasoning, theorem discovery, proof, and LaTeX remain first-class whenever relevant, but non-mathematical work must not be forced into mathematical form. Use internal context as exploration history and independently verify every claim you retain. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -43,22 +44,18 @@ def get_outline_create_system_prompt() -> str: REQUIRED SECTION STRUCTURE (MANDATORY): Your outline MUST include these sections in this exact order: -1. **Abstract** - OPTIONAL (if included, appears first; brief summary written last during construction) +1. **Abstract** - OPTIONAL (if included, appears first, unnumbered; brief summary written last during construction) 2. **Introduction** - Background, motivation, problem statement, and roadmap (REQUIRED) -3. **Body Sections** - Main content (numbered II, III, IV, etc.) covering: - - Preliminaries/Definitions - - Main Results/Theorems - - Proofs - - Additional analysis as needed +3. **Body Sections** - Main solution content (numbered II, III, IV, etc.) using the structure appropriate to the objective. Examples include definitions → theorems → proofs for mathematics; constraints → architecture/algorithm → correctness/interfaces/complexity → evaluation for software; requirements → mechanism/design → tradeoffs/failure modes → verification for engineering; hypothesis/model → evidence/protocol → falsifiable predictions/analysis/limitations for empirical work; or objective → constraints/causal model → intervention/risks → evaluation for strategy or policy. 4. **Conclusion** - Summary of findings and implications (always the LAST content section) (REQUIRED) STRICT NAMING REQUIREMENTS: -- The section named "Abstract" is OPTIONAL - if included, can use "Abstract", "I. Abstract", or "0. Abstract" +- The section named "Abstract" is OPTIONAL - if included, it MUST use the unnumbered heading "Abstract" - The section named "Introduction" MUST use exactly that word: "Introduction" (or "I. Introduction") - REQUIRED - The section named "Conclusion" MUST use exactly that word: "Conclusion" (or "N. Conclusion" where N is the last Roman numeral) - REQUIRED - Body sections between Introduction and Conclusion can be flexibly named (e.g., "II. Preliminaries", "III. Main Results") -🔍 CORRECT OUTLINE FORMATS (THREE VALID OPTIONS): +🔍 CORRECT OUTLINE FORMATS (TWO VALID OPTIONS): **Option 1 - With Abstract (unnumbered, recommended):** @@ -87,35 +84,7 @@ def get_outline_create_system_prompt() -> str: B. Historical significance ``` -**Option 2 - With Abstract (numbered):** - -``` -I. Abstract - -II. Introduction - A. Historical context - ... - -III. Preliminaries - ... - -VI. Conclusion - ... -``` - -OR with zero-based numbering: - -``` -0. Abstract - -I. Introduction - ... - -V. Conclusion - ... -``` - -**Option 3 - Without Abstract (also valid):** +**Option 2 - Without Abstract (also valid):** ``` I. Introduction @@ -162,12 +131,13 @@ def get_outline_create_system_prompt() -> str: - Organize the paper around the strongest rigorous direct answer the paper can justify - Prefer sections that directly solve, partially solve, refute, or sharply constrain the user's question over broad background accumulation - Include background and preliminaries only to the extent needed to support the direct answer cleanly and rigorously +- Do not require theorem/proof structure for an objective that is better served by another rigorous, independently checkable solution form - Produce a numbered outline with major sections and subsections - Incorporate the strongest helpful source ideas where appropriate - Flag gaps explicitly if the evidence is insufficient - Reference supporting content from the aggregator database where appropriate, but do not mirror it mechanically -- Ensure outline supports a coherent, logical flow for the final mathematical document +- Ensure the outline supports a coherent, logical flow for the final solution document - Replace unsupported empirical result claims with neutral headings such as "Evaluation Plan", "Proposed Validation", or "Expected Runtime Benefits" ITERATIVE REFINEMENT PROCESS: @@ -187,8 +157,8 @@ def get_outline_create_system_prompt() -> str: WHEN TO MARK outline_complete=true (LOCK OUTLINE): - The outline makes strong use of any source material it chooses to use and does not omit clearly crucial material for its chosen scope -- Required sections (Abstract, Introduction, Body, Conclusion) present with exact names -- Sections follow logical mathematical progression (definitions → theorems → proofs) +- Required sections (Introduction, Body, Conclusion) are present with exact names; Abstract is optional in the outline and required in the final paper +- Sections follow the domain-appropriate solution progression promised by the paper; mathematical work should use definitions → theorems → proofs when that is the strongest route - The outline optimally serves the paper title and user's compiler-directing prompt - The outline is focused on the strongest rigorous direct answer available, without unnecessary detours - No further refinement would meaningfully improve the outline @@ -222,19 +192,19 @@ def get_outline_create_system_prompt() -> str: - The outline MUST include: Introduction, at least one Body section, and Conclusion (Abstract is optional) - Clearly crucial source material for the chosen scope should have a place in the outline, but you do NOT need to mirror every brainstorm/database entry - The outline should support a coherent, logical flow for the final document -- Sections should build upon each other logically (definitions → theorems → proofs) +- Sections should build upon each other through a domain-appropriate solution route; for mathematical work, retain definitions → theorems → proofs when appropriate - The outline should align with the user's compiler-directing prompt goals - The outline should prioritize the strongest direct rigorous route to answering the prompt - DO NOT include a separate References or Citations section in the outline -- All content must be rooted in sound mathematical reasoning; aggregator/brainstorm material is optional support, not a mandatory checklist +- All content must use claim-type and domain-appropriate rigor; aggregator/brainstorm material is optional support, not a mandatory checklist - NO unfounded claims or logical fallacies -- Focus on rigorous mathematical arguments +- Focus on an independently checkable, direct answer-bearing solution route rather than a broad survey - DO NOT include unsupported numeric empirical claims in section or subsection headings The validator will REJECT your outline if: - Missing required sections: Introduction or Conclusion - Section names don't match exactly (e.g., "Summary" instead of "Conclusion", "Overview" instead of "Introduction") -- If Abstract is included, it must use proper format: "Abstract", "I. Abstract", or "0. Abstract" (not descriptive text) +- If Abstract is included, it must use the unnumbered heading "Abstract" (not numbered or descriptive text) - Sections are out of order (e.g., Conclusion before body sections) - No body sections between Introduction and Conclusion - The outline includes unsupported benchmark-style numbers or hardware results as if already established @@ -242,7 +212,7 @@ def get_outline_create_system_prompt() -> str: CRITICAL - HOW TO FIX COMMON REJECTION: If validator says "MISSING_REQUIRED_SECTION: Introduction", ensure you have a line with "Introduction" or "I. Introduction". -Abstract is OPTIONAL - you can include it ("Abstract", "I. Abstract", or "0. Abstract") or omit it entirely. +Abstract is OPTIONAL - you can include it as the unnumbered heading "Abstract" or omit it entirely. Output your response ONLY as JSON in this exact format: { @@ -269,7 +239,6 @@ def get_outline_create_system_prompt() -> str: Your outline can start with Abstract (optional) or Introduction (required). Examples: - With Abstract: "content": "Abstract\\n\\nI. Introduction\\n..." -- With numbered Abstract: "content": "I. Abstract\\n\\nII. Introduction\\n..." - Without Abstract: "content": "I. Introduction\\n..." """ @@ -295,7 +264,7 @@ def get_outline_update_system_prompt() -> str: """ + OUTLINE_EMPIRICAL_PROVENANCE_RULES + """ - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been proven correct. Aggressively preserve or strengthen the strongest credible and genuinely novel solution to the user's exact objective. Use claim-type and domain-appropriate rigor; mathematical reasoning, theorems, proofs, and LaTeX remain first-class when relevant. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -303,7 +272,7 @@ def get_outline_update_system_prompt() -> str: REQUIRED SECTION STRUCTURE (MUST BE PRESERVED): The outline MUST maintain these exact sections in this exact order: -1. **Abstract** - Brief summary (exactly "Abstract") +1. **Abstract** - OPTIONAL in the outline; if present, it appears first and uses the unnumbered heading "Abstract". It is required in the final paper and written last. 2. **Introduction** - Background and roadmap (exactly "Introduction" or "I. Introduction") 3. **Body Sections** - Main content (numbered II, III, IV, etc.) 4. **Conclusion** - Summary of findings (exactly "Conclusion" or "N. Conclusion") @@ -357,9 +326,9 @@ def get_outline_update_system_prompt() -> str: The validator checks YOUR SUBMISSION for placeholder text, not the existing outline structure. CRITICAL REQUIREMENTS FOR UPDATES: -- All added content must be rooted in sound mathematical reasoning; source database material is optional support, not a mandatory checklist +- All added content must use claim-type and domain-appropriate rigor; source database material is optional support, not a mandatory checklist - NO unfounded claims or logical fallacies -- Focus on rigorous mathematical arguments +- Focus on additions that strengthen the direct answer-bearing solution route; retain theorem/proof structure for mathematical work when relevant - Prefer additions that materially strengthen the paper's direct answer rather than merely broadening coverage - NEVER change the names of Abstract, Introduction, or Conclusion sections - New body sections must be inserted between Introduction and Conclusion @@ -373,8 +342,7 @@ def get_outline_update_system_prompt() -> str: 4. Use operation="insert_after" to add after an anchor point OPERATIONS: -- "insert_after": Find old_string exactly (anchor), insert new_string after it (most common for adding sections) -- "replace": Find old_string exactly, replace with new_string (for fixing section names) +- "insert_after": Find old_string exactly (anchor), insert new_string after it UNIQUENESS REQUIREMENT: - old_string MUST be unique in the outline @@ -385,7 +353,7 @@ def get_outline_update_system_prompt() -> str: Output your response ONLY as JSON in this exact format: { "needs_update": true or false, - "operation": "insert_after | replace", + "operation": "insert_after", "old_string": "exact text from outline to find (anchor point, empty if needs_update=false)", "new_string": "new sections/subsections to add (empty if needs_update=false)", "reasoning": "Why addition is or isn't needed" @@ -424,15 +392,15 @@ def get_outline_json_schema() -> str: REQUIRED JSON FORMAT (for outline update): { "needs_update": true OR false, - "operation": "insert_after | replace", + "operation": "insert_after", "old_string": "exact text from outline (anchor point, empty if needs_update=false)", "new_string": "new sections to add (empty if needs_update=false)", "reasoning": "string - explanation of decision" } MANDATORY SECTION STRUCTURE: -Every outline MUST contain these sections with EXACT names in this order: -1. Abstract (exactly "Abstract") +Every outline MUST contain the required sections below in this order; Abstract is optional in the outline but required in the final paper: +1. Abstract (optional; if included, use the unnumbered heading "Abstract") 2. Introduction (exactly "Introduction" or "I. Introduction") 3. Body sections (II, III, IV, etc. - flexible naming) 4. Conclusion (exactly "Conclusion" or "N. Conclusion") diff --git a/backend/compiler/prompts/review_prompts.py b/backend/compiler/prompts/review_prompts.py index 0c2ca1b..bb63afe 100644 --- a/backend/compiler/prompts/review_prompts.py +++ b/backend/compiler/prompts/review_prompts.py @@ -1,19 +1,22 @@ """ -Review prompts for mathematical document cleanup and error correction. +Review prompts for rigorous solution-document cleanup and error correction. """ from backend.compiler.memory.compiler_rejection_log import compiler_rejection_log EMPIRICAL_PROVENANCE_REVIEW_RULES = """EMPIRICAL PROVENANCE AND CITATION RULES: -- Classify substantive claims as one of: theoretical claim, literature claim, empirical claim, or artifact claim. -- Theoretical claims must be supported by sound derivation, proof, or explicit assumptions inside the paper. +- Apply claim-type and domain-appropriate rigor to every substantive claim. +- Mathematical claims require sound derivation, proof, or explicit assumptions inside the paper. +- Engineering and software claims require mechanisms, constraints, interfaces, failure modes, feasibility reasoning, and verification plans appropriate to the claim. +- Strategic or causal claims require valid inference, explicit assumptions, and realistic limitations. - Literature claims must include explicit in-text citations identifying the external source. Do NOT rely on vague phrases like "studies show" or "the literature suggests". - Empirical claims include benchmarks, latency, throughput, speedups, accuracy, perplexity, ablations, wall-clock measurements, hardware utilization numbers, and dataset/task results. - Artifact claims include statements about code, kernels, measurements, experiments, benchmark logs, reproductions, or "accompanying" implementations. - Empirical or artifact claims are acceptable ONLY if they are backed by an explicit external citation or by a provided artifact in context. If not backed, they must be removed or rewritten as hypotheses, design goals, expected benefits, proposed experiments, or future work. - NEVER invent citations, experiments, benchmark numbers, hardware measurements, datasets, or code artifacts. -- If external information is retained, it must remain explicitly cited in-text. Do NOT imply that unsupported facts were externally verified.""" +- If external information is retained, it must remain explicitly cited in-text. Do NOT imply that unsupported facts were externally verified. +- Novelty never overrides correctness, provenance, safety, or honesty.""" EMPIRICAL_RED_TEAM_REVIEW_FOCUS = """PRE-ABSTRACT EMPIRICAL RED-TEAM TASK: @@ -23,6 +26,7 @@ - unsupported benchmark numbers - uncited external results - benchmark-shaped claims presented as established facts +- unsupported claims that a proposed implementation, mechanism, or evaluation has already been built or completed Inspect especially for: - speedup, latency, throughput, bandwidth, utilization, clock-cycle, memory, or hardware claims @@ -40,7 +44,7 @@ def get_review_system_prompt() -> str: """Get system prompt for document review/cleanup mode.""" - return """You are reviewing the current mathematical document draft for errors and needed improvements. Your role is to: + return """You are reviewing the current rigorous, solution-oriented research paper or report for errors and needed improvements. Your role is to: 1. Review ONLY the current document (aggregator database is NOT in your context for this task) 2. Identify any obvious errors or issues @@ -59,7 +63,7 @@ def get_review_system_prompt() -> str: """ + EMPIRICAL_PROVENANCE_REVIEW_RULES + """ - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what has been explored by AI agents, NOT what has been proven correct. Review against the user's exact objective using claim-type and domain-appropriate rigor. Mathematical accuracy and proof gaps remain mandatory checks when applicable; non-mathematical work must not be rejected merely for lacking mathematical form. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -69,8 +73,9 @@ def get_review_system_prompt() -> str: Review the document for these specific issues: - Grammar errors - Clarity problems -- Mathematical accuracy issues +- Domain-appropriate correctness issues, including mathematical accuracy or proof gaps when applicable - Logical errors or gaps +- Missing mechanisms, constraints, interfaces, feasibility reasoning, implementation detail, evaluation plans, or failure modes when the paper's claims require them - Structural issues - Redundancy - Forward-looking structural previews @@ -103,21 +108,21 @@ def get_review_system_prompt() -> str: - For replace, prefer editable content only; if a protected marker is accidentally included as trailing context, validation may trim it - Do NOT include any of these markers in new_string / generated edit content - Placeholders in the current document are expected - don't try to remove them -- Your edits should contain only actual mathematical prose +- Your edits should contain only actual paper prose; mathematical notation and LaTeX remain valid when relevant WHEN TO MAKE AN EDIT: - Clear grammatical errors - Obvious redundancy that should be removed - Coherence issues between sections - Terminology inconsistencies -- Mathematical inaccuracies or logical errors +- Domain-specific inaccuracies, mathematical/proof errors when applicable, or logical gaps - Significant clarity improvements possible - Forward-looking structural language outside introduction (e.g., 'Section III will...', bulleted lists of future content) - Unfounded claims or logical fallacies that should be corrected - Unsupported empirical claims, unsupported artifact/code claims, or uncited literature claims - Numeric benchmark-style claims in narrative text that are not explicitly sourced - Statements implying experiments, measurements, or implementations that are not actually evidenced -- Generic exploratory wording that obscures a stronger justified direct answer already present in the draft +- Generic exploratory or conventional exposition that obscures a stronger supported novel route or justified direct answer WHEN NOT TO MAKE AN EDIT: - Document is acceptable for a draft in progress @@ -204,7 +209,7 @@ def get_review_json_schema() -> str: "operation": "replace", "old_string": "", "new_string": "", - "reasoning": "The document is coherent and mathematically accurate for its current stage of construction. All proofs are logically sound and definitions are properly introduced before use. No immediate corrections required." + "reasoning": "The document is coherent, directly serves the user's objective, and applies the appropriate rigor to its claims. No substantive correctness, provenance, structural, feasibility, or verification issue warrants an edit at this stage." } Example (Deletion - removing redundant content): diff --git a/backend/compiler/validation/compiler_validator.py b/backend/compiler/validation/compiler_validator.py index 69d4e90..b23a61d 100644 --- a/backend/compiler/validation/compiler_validator.py +++ b/backend/compiler/validation/compiler_validator.py @@ -1,7 +1,6 @@ """ Compiler validator - validates document edits for coherence, rigor, and placement. """ -import json import logging import uuid from typing import Optional, Dict, Any, Callable, Tuple @@ -12,7 +11,6 @@ from backend.shared.json_parser import parse_json, sanitize_model_output_for_retry_context from backend.shared.response_extraction import extract_message_text from backend.shared.utils import count_tokens -from backend.aggregator.validation.json_validator import json_validator from backend.compiler.memory.paper_memory import ( paper_memory, PAPER_ANCHOR, @@ -24,15 +22,30 @@ logger = logging.getLogger(__name__) -CLAIM_TYPE_VALIDATION_RULES = """CLAIM TYPE VALIDATION (CRITICAL): -- Classify substantive claims as one of: theoretical claim, literature claim, empirical claim, or artifact claim. -- Theoretical claims are acceptable only if they are supported by sound derivation, proof, or explicit assumptions in the document. -- Literature claims are acceptable only if they include explicit in-text citations identifying the external source. Vague phrases like "studies show" are insufficient. -- Empirical claims include benchmark numbers, latency, throughput, speedups, accuracy, perplexity, ablations, hardware metrics, measured runtimes, and evaluation outcomes. -- Artifact claims include statements about code, kernels, experiments, benchmark logs, reproductions, or accompanying implementations. -- Empirical or artifact claims may be presented as established fact ONLY when backed by an explicit external citation or a provided artifact in context. -- If empirical or artifact support is absent, acceptable wording is limited to hypothesis, expected benefit, design target, proposed experiment, validation plan, limitation, or future work. -- Reject content that invents citations, experiments, benchmark numbers, hardware measurements, datasets, or code artifacts.""" +CLAIM_TYPE_VALIDATION_RULES = """CLAIM-TYPE AND DOMAIN-APPROPRIATE VALIDATION (CRITICAL): +- Classify each substantive claim by the standard that can actually establish it; a submission may contain several claim types. +- Mathematical/theoretical claims require a sound derivation or proof, or clearly stated assumptions and a correctly delimited conjectural/conditional status. Check definitions, quantifiers, edge cases, and inferential steps when applicable. +- Literature/historical claims require an explicit in-text citation identifying the external source. Vague authority phrases such as "studies show" are insufficient, and citations must not be invented. +- Empirical claims include measured outcomes, benchmark numbers, latency, throughput, speedups, accuracy, perplexity, ablations, hardware metrics, datasets, and experimental observations. Established empirical claims require an explicit external citation or a provided artifact; otherwise they must be framed as hypotheses, predictions, targets, protocols, or proposed evaluations. +- Artifact claims include statements that code, kernels, experiments, logs, datasets, reproductions, or implementations exist or have particular behavior. They require a cited or provided inspectable artifact; a proposed artifact must be labeled as proposed. +- Engineering/software claims require a technically coherent mechanism, explicit requirements and constraints, compatible interfaces, feasible resource assumptions, relevant complexity or performance reasoning, failure-mode analysis, and a verification/evaluation route. A design is not validated merely because it is plausible prose. +- Strategic/policy/causal claims require an explicit objective, actors and constraints, a defensible causal model, intervention mechanism, assumptions, countervailing effects and risks, and measurable evaluation or falsification criteria. Correlation, intention, or narrative plausibility alone does not establish causation. +- Accept mixed or other domain claims under the strongest applicable standard: factual accuracy, logical validity, methodological soundness, provenance, feasibility, and independent checkability. +- Reject fabricated evidence, citations, experiments, metrics, datasets, implementations, capabilities, guarantees, or causal certainty.""" + +MATHEMATICAL_RIGOR_CLAIM_RULES = """CLAIM TYPE VALIDATION (CRITICAL): +- Mathematical and theoretical claims are acceptable only when supported by sound derivation, proof, or explicit assumptions in the document. +- Literature claims require explicit in-text citations identifying the external source; vague authority claims are insufficient. +- Empirical or artifact claims may be stated as established fact only when backed by an explicit citation or provided artifact. +- Without that support, empirical or artifact material must remain a hypothesis, expected benefit, proposed validation, limitation, or future work. +- Reject invented citations, experiments, metrics, datasets, code artifacts, or mathematical support.""" + +PRIMARY_VALIDATION_FINAL_REMINDER = ( + "\n\nFINAL RESPONSE REQUIREMENT: Return exactly one JSON object matching " + "the primary validation schema above. The primary decision and required " + "boolean checks are authoritative; any optional advisory extension must " + "not replace, weaken, or contradict them." +) def _diagnostic_char_info(text: str, max_chars: int = 100) -> str: @@ -340,13 +353,8 @@ def find_with_normalized_hyphens(needle: str, haystack: str) -> Tuple[int, str]: return (-1, "") -class JSONParseError(Exception): - """Raised when JSON parsing fails - signals need for retry""" - def __init__(self, message: str, response: str, parse_error: Exception): - self.message = message - self.response = response - self.parse_error = parse_error - super().__init__(message) +class ValidatorContractError(ValueError): + """Raised only when model output violates the validator JSON contract.""" class CompilerValidator: @@ -364,6 +372,7 @@ def __init__( user_prompt: str, websocket_broadcaster: Optional[Callable] = None, proof_database_store=None, + solution_path_manager=None, ): self.model_name = model_name self.user_prompt = ( @@ -372,6 +381,7 @@ def __init__( else user_prompt ) self.websocket_broadcaster = websocket_broadcaster + self.solution_path_manager = solution_path_manager self._initialized = False # Task tracking for workflow panel and boost integration @@ -396,60 +406,46 @@ async def initialize(self) -> None: logger.info(f"Compiler validator initialized with model: {self.model_name}") - def _parse_json_response(self, response: str) -> Dict[str, Any]: - """ - Parse JSON response using central parse_json utility. - Raises JSONParseError on failure to signal retry needed. - - The central parse_json() handles: - - Reasoning tokens (<think>...</think>) - - Markdown code blocks (```json ... ```) - - Control tokens - - LaTeX escape sequences - - Malformed JSON detection - - Enhanced error logging - """ - try: - # Use central parse_json for consistent sanitization and parsing - return parse_json(response) - except (json.JSONDecodeError, ValueError) as e: - # Raise custom exception to signal retry needed - logger.error(f"Compiler validator: JSON parse failed - {e}") - raise JSONParseError(f"Failed to parse JSON response", response, e) - - def _fallback_parse(self, response: str) -> Dict[str, Any]: - """ - Fallback parser for when all JSON parse attempts fail. - Uses keyword heuristics to extract decision from natural language. - - Strategy: Look for "accept" vs "reject" keywords in response. + @staticmethod + def _validate_primary_validation_contract( + parsed: Any, + *, + require_checks: bool, + ) -> Dict[str, Any]: + """Validate only the primary response fields used by the compiler. + + Optional extensions such as ``solution_path_update`` deliberately remain + outside this contract so their omission or malformed content cannot alter + an otherwise valid primary decision. """ - response_lower = response.lower() - - # Detect decision via keywords - decision = "reject" # Default to reject for safety - if 'accepted: true' in response_lower or 'decision: accept' in response_lower: - decision = "accept" - elif 'accepted: false' in response_lower or 'decision: reject' in response_lower: - decision = "reject" - elif 'accept' in response_lower and 'reject' not in response_lower: - decision = "accept" - elif 'reject' in response_lower: - decision = "reject" - - logger.warning(f"CompilerValidator: Fallback parser used (determined decision={decision} from keywords)") - - return { - 'decision': decision, - 'reasoning': response, # Full response, no truncation - } + if not isinstance(parsed, dict): + raise ValidatorContractError("Validation response must be a JSON object") + + decision = parsed.get("decision") + if decision not in {"accept", "reject"}: + raise ValidatorContractError( + "decision must be exactly 'accept' or 'reject'" + ) + + reasoning = parsed.get("reasoning") + if not isinstance(reasoning, str) or not reasoning.strip(): + raise ValidatorContractError("reasoning must be a non-empty string") + + if require_checks: + for field in ("coherence_check", "rigor_check", "placement_check"): + if type(parsed.get(field)) is not bool: + raise ValidatorContractError(f"{field} must be a boolean") + + return parsed async def _parse_json_with_retry( self, response: str, original_prompt: str, system_prompt: str, - retry_count: int = 0 + retry_count: int = 0, + *, + require_checks: bool = True, ) -> Dict[str, Any]: """ Parse JSON with a single conversational retry on failure. @@ -461,7 +457,7 @@ async def _parse_json_with_retry( Flow: 1. Attempt parse via parse_json() 2. If fails: ask LLM to reformat via conversation (single attempt) - 3. If still fails: use _fallback_parse() + 3. If still invalid: raise a contract error for structured rejection Args: response: LLM response to parse @@ -470,20 +466,25 @@ async def _parse_json_with_retry( retry_count: Current retry attempt (0-indexed) - kept for API compatibility Returns: - Parsed dict (from JSON or fallback) + Parsed and structurally valid primary response dict """ # First attempt: try to parse JSON directly try: parsed = parse_json(response) - return parsed + return self._validate_primary_validation_contract( + parsed, + require_checks=require_checks, + ) except Exception as parse_error: logger.warning(f"CompilerValidator: JSON parse failed, attempting single retry: {parse_error}") - # If we're already in a retry, use fallback immediately to prevent deep recursion + # If already in a retry, fail closed instead of inferring a decision + # from malformed prose. if retry_count > 0: - logger.info("CompilerValidator: Already in retry, using fallback parser") - return self._fallback_parse(response) + raise ValidatorContractError( + "Validation JSON contract remained invalid after retry" + ) from parse_error # Build retry prompt asking for reformatted JSON. Keep failed-output # context, but sanitize it before any replay in prompt or assistant turn. @@ -497,6 +498,8 @@ async def _parse_json_with_retry( f"YOUR PREVIOUS RESPONSE:\n{failed_output_preview}\n\n" f"PARSE ERROR: {str(parse_error)}\n\n" "Please provide the exact same validation decision in valid JSON format.\n" + "Retain the one valid optional top-level `solution_path_update` from " + "your previous response if present; omission remains valid.\n" "CRITICAL: Properly escape backslashes (use \\\\) and quotes (use \\\").\n" "Respond with ONLY the corrected JSON, no explanation." ) @@ -544,24 +547,32 @@ async def _parse_json_with_retry( if not retry_response.get("choices"): logger.error("CompilerValidator: Retry request returned no choices") - return self._fallback_parse(response) + raise ValidatorContractError( + "Validation JSON retry returned no choices" + ) message = retry_response["choices"][0]["message"] retry_output = extract_message_text(message) try: parsed = parse_json(retry_output) + parsed = self._validate_primary_validation_contract( + parsed, + require_checks=require_checks, + ) logger.info("CompilerValidator: Retry succeeded!") return parsed except Exception as retry_parse_error: logger.warning(f"CompilerValidator: Retry parse failed: {retry_parse_error}") - return self._fallback_parse(response) + raise ValidatorContractError( + "Validation JSON contract remained invalid after bounded retry" + ) from retry_parse_error except RetryableProviderError: raise except Exception as retry_error: logger.error(f"CompilerValidator: Retry request failed - {retry_error}") - return self._fallback_parse(response) + raise def _handle_protected_marker_old_string( self, @@ -708,6 +719,37 @@ def _pre_validate_exact_string_match( json_valid=True, validation_stage="pre-validation" ) + + if ( + submission.operation == "full_content" + and not document_is_empty + and submission.mode != "outline_create" + ): + logger.warning( + "Pre-validation failed: full_content cannot replace a non-empty %s", + document_name, + ) + return CompilerValidationResult( + submission_id=submission.submission_id, + decision="reject", + reasoning=( + "NON_EMPTY_FULL_CONTENT_ERROR: operation='full_content' is only " + f"valid when the {document_name} is empty. Replacing a populated " + f"{document_name} would discard existing work.\n\n" + "FIX REQUIRED:\n" + "1. Use replace, insert_after, or delete with an exact old_string.\n" + "2. Preserve all existing sections and system-managed markers." + ), + summary=( + f"full_content cannot replace a non-empty {document_name} " + "(pre-validation)" + ), + coherence_check=True, + rigor_check=True, + placement_check=False, + json_valid=True, + validation_stage="pre-validation", + ) # Only check operations that require old_string matching if submission.operation not in ("replace", "insert_after", "delete"): @@ -1015,6 +1057,7 @@ async def validate_submission( CompilerValidationResult """ logger.info(f"Validating {submission.mode} submission: {submission.submission_id}") + validation_mode = self._get_effective_validation_mode(submission) # PRE-PROCESSING: Strip any placeholder text from submission content and new_string # Instead of rejecting, we silently strip placeholders to simplify the workflow @@ -1057,7 +1100,16 @@ async def validate_submission( return string_match_result # Build validation prompt - prompt = self._build_validation_prompt(submission, current_paper, current_outline) + prompt = self._build_validation_prompt( + submission, + current_paper, + current_outline, + validation_mode=validation_mode, + ) + if validation_mode != "rigor_lean_placement": + from backend.shared.solution_path.integration import with_validator_hook + prompt = with_validator_hook(prompt, self.solution_path_manager) + prompt += PRIMARY_VALIDATION_FINAL_REMINDER # CRITICAL: Verify actual prompt size fits in context window from backend.shared.utils import count_tokens @@ -1119,13 +1171,31 @@ async def validate_submission( llm_output = extract_message_text(message) # Parse JSON response with retry logic - validation_data = await self._parse_json_with_retry( - llm_output, - prompt, - "", # system_prompt not used in this call - 0 # initial retry count - ) - + try: + validation_data = await self._parse_json_with_retry( + llm_output, + prompt, + "", # system_prompt not used in this call + 0, # initial retry count + ) + except ValidatorContractError as contract_error: + logger.warning( + "CompilerValidator: rejecting structurally invalid validation response: %s", + contract_error, + ) + if self.task_tracking_callback: + self.task_tracking_callback("completed", task_id) + return CompilerValidationResult( + submission_id=submission.submission_id, + decision="reject", + reasoning=f"Invalid validator JSON contract: {contract_error}", + summary="Invalid validator JSON contract", + coherence_check=False, + rigor_check=False, + placement_check=False, + json_valid=False, + validation_stage="llm_validation", + ) # Extract validation decision decision = validation_data.get("decision", "reject") reasoning = validation_data.get("reasoning", "No reasoning provided") @@ -1139,16 +1209,46 @@ async def validate_submission( # of truth for mathematical rigor. Force rigor_check=True regardless # of what the LLM emitted so the criterion is never the reason for # a rejection on this kind of submission. - if ( - submission.mode == "rigor" - and (submission.metadata or {}).get("rigor_mode") == "lean_placement" - ): + if validation_mode == "rigor_lean_placement": if not rigor: logger.info( "Validator returned rigor_check=False for lean_placement submission; " "forcing True because Lean 4 verified the math." ) rigor = True + + if decision == "accept" and not (coherence and rigor and placement): + failed_checks = [ + name + for name, passed in ( + ("coherence_check", coherence), + ("rigor_check", rigor), + ("placement_check", placement), + ) + if not passed + ] + logger.warning( + "CompilerValidator: converting inconsistent acceptance to rejection; " + "failed checks: %s", + ", ".join(failed_checks), + ) + decision = "reject" + reasoning = ( + f"{reasoning}\n\n" + "CONTRACT CONSISTENCY: Acceptance requires every applicable " + f"validation check to pass. Failed: {', '.join(failed_checks)}." + ) + + if validation_mode != "rigor_lean_placement": + from backend.shared.solution_path.integration import enqueue_optional_update + await enqueue_optional_update( + validation_data, + self.solution_path_manager, + proposer_role=self.role_id, + source_task_id=task_id, + source_phase=f"compiler_{submission.mode}_validation", + source_decision=decision, + ) # Create summary for rejection log (max 750 chars) summary = reasoning[:750] @@ -1206,6 +1306,9 @@ async def validate_brainstorm_operation( logger.info(f"Validating brainstorm retroactive operation: {brainstorm_op.action}") prompt = self._build_brainstorm_validation_prompt(brainstorm_op, brainstorm_content) + from backend.shared.solution_path.integration import with_validator_hook + prompt = with_validator_hook(prompt, self.solution_path_manager) + prompt += PRIMARY_VALIDATION_FINAL_REMINDER actual_prompt_tokens = count_tokens(prompt) from backend.shared.config import system_config, rag_config @@ -1257,9 +1360,42 @@ async def validate_brainstorm_operation( message = response["choices"][0]["message"] llm_output = extract_message_text(message) - validation_data = await self._parse_json_with_retry(llm_output, prompt, "", 0) - + try: + validation_data = await self._parse_json_with_retry( + llm_output, + prompt, + "", + 0, + require_checks=False, + ) + except ValidatorContractError as contract_error: + logger.warning( + "CompilerValidator: rejecting structurally invalid brainstorm validation: %s", + contract_error, + ) + if self.task_tracking_callback: + self.task_tracking_callback("completed", task_id) + return CompilerValidationResult( + submission_id=str(uuid.uuid4()), + decision="reject", + reasoning=f"Invalid validator JSON contract: {contract_error}", + summary="Invalid validator JSON contract", + coherence_check=False, + rigor_check=False, + placement_check=False, + json_valid=False, + validation_stage="llm_validation", + ) + from backend.shared.solution_path.integration import enqueue_optional_update decision = validation_data.get("decision", "reject") + await enqueue_optional_update( + validation_data, + self.solution_path_manager, + proposer_role=self.role_id, + source_task_id=task_id, + source_phase="retroactive_brainstorm_validation", + source_decision=decision if decision in {"accept", "reject"} else None, + ) reasoning = validation_data.get("reasoning", "No reasoning provided") result = CompilerValidationResult( @@ -1315,9 +1451,9 @@ def _build_brainstorm_validation_prompt( if action == "delete": system_prompt += """VALIDATION CRITERIA (DELETE): A brainstorm submission should be REMOVED if it: -1. Contains mathematical errors or logically unsound reasoning +1. Contains factual, technical, mathematical, methodological, or logical errors under the standards appropriate to its claims and domain 2. Is redundant with other submissions (content fully covered elsewhere) -3. Contradicts established mathematical principles evident in other submissions +3. Contradicts well-supported constraints, evidence, principles, or verified results evident in other submissions 4. Was marginally useful but provides no unique value given the current database state KEEP the submission if: @@ -1330,10 +1466,10 @@ def _build_brainstorm_validation_prompt( elif action == "edit": system_prompt += """VALIDATION CRITERIA (EDIT): A brainstorm submission edit should be ACCEPTED if: -1. The corrected version fixes a genuine mathematical error +1. The corrected version fixes a genuine factual, technical, mathematical, methodological, or logical error 2. The corrected version is more accurate than the original 3. The correction improves the submission's value to the knowledge pool -4. The correction is mathematically sound and well-justified +4. The correction is sound and well-justified under the rigor appropriate to its domain and claim types REJECT the edit if: 1. The original was not actually wrong @@ -1346,10 +1482,10 @@ def _build_brainstorm_validation_prompt( elif action == "add": system_prompt += """VALIDATION CRITERIA (ADD): A new brainstorm submission should be ACCEPTED if: -1. It adds genuinely new mathematical insight not already in the database +1. It adds a genuinely new, objective-relevant insight, mechanism, design, algorithm, experiment, impossibility argument, implementation path, risk analysis, mathematical result, or other solution contribution not already in the database 2. It connects existing concepts in novel ways -3. It provides concrete methods, theorems, proofs, or techniques -4. It is grounded in established mathematical principles +3. It provides concrete methods, evidence plans, artifacts, theorems, proofs, techniques, or other actionable advances appropriate to the objective +4. Its claims are grounded in appropriate evidence, sound reasoning, explicit assumptions, or rigorous derivation as their types require REJECT the addition if: 1. It is redundant with existing submissions @@ -1436,32 +1572,36 @@ def _strip_placeholder_text(self, text: str) -> str: return result.strip() + @staticmethod + def _get_effective_validation_mode(submission: CompilerSubmission) -> str: + """Resolve the validator mode once for all downstream behavior.""" + if ( + submission.mode == "rigor" + and (submission.metadata or {}).get("rigor_mode") == "lean_placement" + ): + return "rigor_lean_placement" + return submission.mode + def _build_validation_prompt( self, submission: CompilerSubmission, current_paper: str, - current_outline: Optional[str] + current_outline: Optional[str], + *, + validation_mode: Optional[str] = None, ) -> str: """Build validation prompt based on submission mode.""" + validation_mode = validation_mode or self._get_effective_validation_mode(submission) # Route to appropriate system prompt based on mode - if submission.mode in ["outline_create", "outline_update"]: - system_prompt = self._get_outline_validation_system_prompt(submission.mode) + if validation_mode in ["outline_create", "outline_update"]: + system_prompt = self._get_outline_validation_system_prompt(validation_mode) else: - # For rigor submissions backed by a Lean 4 verified theorem, swap in - # the placement-only criteria. Anything else falls through to the - # normal paper-validation prompt for the submission's mode. - rigor_mode_hint = (submission.metadata or {}).get("rigor_mode") - effective_mode = ( - "rigor_lean_placement" - if submission.mode == "rigor" and rigor_mode_hint == "lean_placement" - else submission.mode - ) - system_prompt = self._get_paper_validation_system_prompt(effective_mode) + system_prompt = self._get_paper_validation_system_prompt(validation_mode) parts = [ system_prompt, "\n---\n", - self._get_validation_json_schema(), + self._get_validation_json_schema(validation_mode), "\n---\n", f"USER COMPILER-DIRECTING PROMPT:\n{self.user_prompt}", "\n---\n", @@ -1475,7 +1615,7 @@ def _build_validation_prompt( # For Lean 4 verified theorem placement, surface the Lean certificate # to the validator so it can reference what was verified. metadata = submission.metadata or {} - if submission.mode == "rigor" and metadata.get("rigor_mode") == "lean_placement": + if validation_mode == "rigor_lean_placement": parts.append("LEAN 4 VERIFICATION CERTIFICATE (DO NOT RE-EVALUATE MATH):\n") parts.append(f"Proof ID: {metadata.get('lean_proof_id', 'unknown')}\n") parts.append(f"Theorem statement: {metadata.get('theorem_statement', '')}\n") @@ -1507,7 +1647,7 @@ def _build_validation_prompt( def _get_outline_validation_system_prompt(self, mode: str) -> str: """Get system prompt for OUTLINE validation (outline_create, outline_update).""" - base_prompt = """You are validating a mathematical document outline submission. Your role is to decide if this submission should be ACCEPTED or REJECTED. + base_prompt = """You are validating an outline for a rigorous research paper or solution report. Decide whether the submission should be ACCEPTED or REJECTED according to the user's exact compiler-directing objective and the claim types and domain actually involved. ⚠️ CRITICAL - INTERNAL CONTENT WARNING ⚠️ @@ -1522,7 +1662,7 @@ def _get_outline_validation_system_prompt(self, mode: str) -> str: """ + CLAIM_TYPE_VALIDATION_RULES + """ - The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. + The internal context shows what AI agents explored, NOT what has been established. Treat it as optional working evidence and independently assess the proposed route with domain-appropriate rigor. WHEN IN DOUBT: Verify independently. Do not assume. Do not trust unverified internal context as truth. @@ -1549,26 +1689,26 @@ def _get_outline_validation_system_prompt(self, mode: str) -> str: REQUIRED SECTION STRUCTURE (MANDATORY): Every outline MUST include these sections with these exact names in this exact order: -1. **Abstract** - OPTIONAL - If included, must be named "Abstract", "I. Abstract", or "0. Abstract" (appears first if present) +1. **Abstract** - OPTIONAL - If included, must use the unnumbered heading "Abstract" (appears first if present) 2. **Introduction** - Must be named exactly "Introduction" or "I. Introduction" (REQUIRED) 3. **Body Sections** - At least one body section (II, III, IV, etc.) between Introduction and Conclusion (REQUIRED) 4. **Conclusion** - Must be named exactly "Conclusion" or "N. Conclusion" (always LAST content section) (REQUIRED) -OUTLINE VALIDATION CRITERIA: -1. SECTION_STRUCTURE: MUST include Introduction, at least one Body section, and Conclusion with exact names (Abstract is optional but recommended) -2. SECTION_ORDER: [Abstract →] Introduction → Body sections → Conclusion (this exact order, where Abstract is optional) -3. COHERENCE: Logically structured with clear sections and subsections -4. COMPLETENESS: Makes effective use of any source material it chooses to use and does not omit clearly crucial material for its chosen scope -5. ALIGNMENT: Aligns with user's compiler-directing prompt goals -6. COMPREHENSIVENESS: Provides sufficient detail to guide mathematical document construction -7. MATHEMATICAL PROGRESSION: Body sections follow logical progression (definitions → main results → theorems → proofs) -8. IN-TEXT CITATIONS ONLY: Must NOT include a separate References or Citations section -9. ANCHOR PRESERVATION: Must not attempt to add content after the end-of-outline anchor markers -10. LOGICAL GROUNDING: Outline may draw from aggregator database material, reference papers, or rigorous reasoning, but it must remain grounded in sound mathematical principles and avoid unfounded claims -11. NO PLACEHOLDER TEXT: Must not contain any placeholder markers (e.g., "[HARD CODED PLACEHOLDER FOR...", "[PLACEHOLDER FOR...", "TO BE WRITTEN AFTER..."). Placeholders are structural markers only - all submitted content must be actual outline content. -12. EMPIRICAL PROVENANCE: Must not include unsupported numeric empirical claims in section/subsection headings or outline prose unless explicitly backed by citation or provided artifact support - -**CRITICAL**: Criterion #11 checks the SUBMISSION CONTENT ONLY. The CURRENT OUTLINE may contain system-managed anchor markers (normal and expected). Do NOT reject a submission just because anchor markers exist in the current outline - only reject if the SUBMISSION ITSELF contains placeholder or anchor text (pre-validation at line 326 catches this before you see it). +OUTLINE VALIDATION DIMENSIONS (ALL TEN MUST PASS): +1. COHERENCE: Sections and subsections form a clear, internally consistent argument. +2. DOMAIN-APPROPRIATE CORRECTNESS: The planned route uses the standards and progression appropriate to its claims and domain. +3. DIRECT SOLUTION VALUE: It defines an answer-bearing route to the exact objective rather than a broad survey or easier adjacent problem. +4. PLACEMENT CONTEXT: Each proposed component has a clear position in the required [Abstract →] Introduction → Body → Conclusion order. +5. NON-REDUNDANCY: Sections make distinct necessary contributions without repetitive coverage. +6. OUTLINE ADHERENCE: The submission preserves required section names/order and, for updates, remains compatible with already-constructed content. +7. CLAIM-TYPE PROVENANCE: Planned claims have an independently checkable support route under the claim-type rules above. +8. SPECIFICITY AND ACTIONABILITY: It identifies concrete arguments, mechanisms, tests, decisions, or deliverables sufficient to guide construction. +9. NOVELTY WITHOUT FABRICATION: It aggressively pursues the strongest credible novel route while exposing assumptions and never inventing support. +10. NO PLACEHOLDER OR MARKER VIOLATIONS: It does not include protected text or place content after the outline anchors. + +ADDITIONAL FORMAT RULE: Use in-text citations only; do not create a separate References or Citations section. + +**CRITICAL**: Dimension #10 checks the SUBMISSION CONTENT ONLY. The CURRENT OUTLINE may contain system-managed anchor markers (normal and expected). Do NOT reject a submission just because anchor markers exist in the current outline - only reject if the SUBMISSION ITSELF contains placeholder or anchor text (pre-validation catches this before you see it). SOURCE MATERIAL POLICY: - The aggregator/brainstorm database is optional support, not a mandatory checklist @@ -1576,6 +1716,7 @@ def _get_outline_validation_system_prompt(self, mode: str) -> str: - Do reject if the outline ignores clearly crucial source material in a way that makes its chosen scope weak, incoherent, or misaligned with the user prompt - Accept selective or divergent outline structures when they better serve the user's prompt and remain rigorous - Prefer outlines that organize the strongest rigorous direct answer to the user's prompt, rather than broad exploratory coverage +- Mathematics remains fully supported and should use theorem/proof progression when useful, but do not reject a non-mathematical outline merely because it lacks definitions, theorems, or proofs YOUR TASK: Verify the submission meets ALL criteria above. Accept only if ALL criteria pass. Reject if ANY criterion fails. @@ -1583,11 +1724,11 @@ def _get_outline_validation_system_prompt(self, mode: str) -> str: REJECTION CATEGORIES (provide specific feedback): - MISSING_REQUIRED_SECTION: Missing Introduction or Conclusion (must have both with exact names; Abstract is optional) - INCORRECT_SECTION_ORDER: Sections are out of order (must be: [Abstract →] Introduction → Body → Conclusion, where Abstract is optional) -- INCORRECT_SECTION_NAME: Section names don't match exactly (e.g., "Summary" instead of "Conclusion", "Overview" instead of "Introduction"; if Abstract included, must be "Abstract", "I. Abstract", or "0. Abstract") -- STRUCTURAL: Body sections not in logical mathematical progression order +- INCORRECT_SECTION_NAME: Section names don't match exactly (e.g., "Summary" instead of "Conclusion", "Overview" instead of "Introduction"; if Abstract is included, it must use the unnumbered heading "Abstract") +- STRUCTURAL: Body sections do not follow the domain-appropriate solution progression - INCOMPLETENESS: Missing clearly crucial source material or necessary structure for the chosen scope - MISALIGNMENT: Doesn't serve user's compiler-directing prompt goals -- INSUFFICIENT_DETAIL: Lacks necessary granularity to guide mathematical document construction +- INSUFFICIENT_DETAIL: Lacks necessary granularity to guide rigorous document construction - FORMAT_VIOLATION: Includes separate References/Citations section (NOT allowed) - ANCHOR_VIOLATION: Content placed after outline anchor markers - EMPIRICAL_PROVENANCE: Unsupported benchmark, hardware, or artifact claim presented as established fact @@ -1599,9 +1740,8 @@ def _get_outline_validation_system_prompt(self, mode: str) -> str: 1. **Abstract Header Check (OPTIONAL)**: ✓ VALID: A line containing ONLY "Abstract" (case-insensitive, may have whitespace) ✓ VALID: "Abstract" or "ABSTRACT" or " Abstract " - ✓ VALID: "I. Abstract" or "0. Abstract" (numbered variants also acceptable) ✓ VALID: Omitting Abstract entirely (it's optional - some outlines don't include it) - ❌ INVALID: "Summary of the paper" or "Abstract: This paper" (if including Abstract, use proper format) + ❌ INVALID: "I. Abstract", "0. Abstract", "Summary of the paper", or "Abstract: This paper" (if including Abstract, use the unnumbered heading) 2. **Introduction Header Check (REQUIRED)**: ✓ VALID: "Introduction" or "I. Introduction" (case-insensitive) @@ -1633,11 +1773,11 @@ def _get_outline_validation_system_prompt(self, mode: str) -> str: mode_specific = { "outline_create": """MODE-SPECIFIC CRITERIA (Outline Creation): - Outline MUST include: Introduction, at least one Body section, Conclusion (Abstract is optional) -- Section names MUST match exactly: "Introduction", "Conclusion" (if Abstract included: "Abstract", "I. Abstract", or "0. Abstract") +- Section names MUST match exactly: "Introduction", "Conclusion" (if Abstract is included: unnumbered "Abstract") - Outline makes effective use of any helpful aggregator database content when relevant, but need not mirror all entries -- Outline provides clear structure for mathematical document construction +- Outline provides a clear, direct, answer-bearing structure for rigorous document construction - Outline aligns with user's compiler-directing prompt -- Body sections follow logical mathematical progression +- Body sections follow the domain-appropriate solution progression; mathematical work retains definitions/results/theorems/proofs when relevant - Outline is comprehensive enough to guide entire exposition FEEDBACK FOR ITERATIVE REFINEMENT: @@ -1670,27 +1810,28 @@ def _get_outline_validation_system_prompt(self, mode: str) -> str: FEEDBACK QUALITY EXAMPLES: ✅ GOOD (Specific and Actionable): -"REJECTION REASON: MISSING_REQUIRED_SECTION - Abstract +"REJECTION REASON: MISSING_REQUIRED_SECTION - Introduction VALIDATION CRITERIA FAILED: #1 SECTION_STRUCTURE WHAT I SAW: -Your outline starts with: 'Summary of the paper's core contribution...' +Your outline starts directly with: 'II. Core Mechanism' WHAT I EXPECTED: -First line: 'Abstract' (just this word, no descriptive text) +An Introduction section before the body sections. FIX REQUIRED: -1. Remove the descriptive text from line 1 -2. Replace it with ONLY the word 'Abstract' -3. The outline lists section names, not content +1. Add 'I. Introduction' before 'II. Core Mechanism' +2. Keep the body sections after Introduction +3. Keep Conclusion as the final required core section EXAMPLE: -Abstract - I. Introduction A. Historical context - ..." + ... +II. Core Mechanism + ... +III. Conclusion" ❌ BAD (Vague): "The outline needs more detail in the main results section." @@ -1716,7 +1857,7 @@ def _get_outline_validation_system_prompt(self, mode: str) -> str: - Update maintains logical progression consistency with existing outline structure - Update doesn't require modification of already-constructed document content - Changes are substantive, not cosmetic -- New sections maintain mathematical document ordering +- New sections maintain the domain-appropriate argument and solution ordering - Update does NOT add a References or Citations section CRITICAL ADDITIVITY CHECK: @@ -1734,7 +1875,123 @@ def _get_outline_validation_system_prompt(self, mode: str) -> str: return base_prompt + mode_specific.get(mode, mode_specific["outline_create"]) def _get_paper_validation_system_prompt(self, mode: str) -> str: - """Get system prompt for DOCUMENT validation (construction, review, rigor).""" + """Route ordinary writing and protected rigor modes to isolated prompts.""" + if mode == "rigor": + return self._get_mathematical_rigor_validation_system_prompt(mode) + if mode == "rigor_lean_placement": + return self._get_lean_placement_validation_system_prompt() + + base_prompt = """You are validating an exact-edit submission to a rigorous research paper or solution report. Decide whether it should be ACCEPTED or REJECTED according to the user's exact compiler-directing objective and the standards appropriate to the claims and domain actually present. + +⚠️ CRITICAL - INTERNAL CONTENT WARNING ⚠️ + +ALL supplied brainstorm databases, accepted submissions, papers, references, outlines, and prior document content are AI-generated working evidence. They are not authoritative merely because this system generated or accepted them. Independently check claims under the strongest applicable standard, do not invent support, and use internal context selectively when it improves the direct solution. + +""" + CLAIM_TYPE_VALIDATION_RULES + """ + +ORDINARY VALIDATION DIMENSIONS (ALL MUST PASS): +1. COHERENCE: The edit is clear, internally consistent, and preserves holistic document flow. +2. DOMAIN-APPROPRIATE CORRECTNESS: The edit satisfies the applicable mathematical, logical, factual, technical, methodological, engineering, empirical, or causal standard. The `rigor_check` JSON field reports this applicable rigor; it does not require mathematics when mathematics is irrelevant. +3. DIRECT SOLUTION VALUE: The edit materially advances the strongest credible answer or solution route to the user's exact objective, rather than adding a broad survey, decorative exposition, or an easier adjacent task. +4. PLACEMENT CONTEXT: The content fits naturally at the pre-validated exact-edit location and at the current construction stage. +5. NON-REDUNDANCY: The edit adds necessary value without repeating existing content or duplicating an existing section or subsection header in the current document. +6. OUTLINE ADHERENCE: The edit serves the promised solution structure and scope. Selective departure from source material is allowed; conflict with the operative outline is not. +7. CLAIM-TYPE PROVENANCE: Every substantive claim satisfies the claim-type rules above, with uncertainty and proposals labeled honestly. +8. SPECIFICITY AND ACTIONABILITY: The content states concrete mechanisms, arguments, requirements, tests, implications, or next inferential steps at the level needed for independent assessment. +9. NOVELTY WITHOUT FABRICATION: Prefer a genuinely stronger or less conventional supported route when available, but reject novelty obtained through unsupported claims, invented evidence, or concealed assumptions. +10. MARKER AND PLACEHOLDER INTEGRITY: Submission content contains no protected marker or placeholder and does not place content beyond the document boundary. + +CLAIM-SENSITIVE SOLUTION FORMS: +- Mathematics may use definitions, lemmas, theorems, derivations, proofs, and corollaries; unsupported theorem claims fail applicable rigor. +- Algorithms/software may use constraints, architecture, algorithms, correctness arguments, interfaces, complexity, implementation details, and evaluation plans. +- Engineering may use requirements, mechanisms, designs, tradeoffs, feasibility, failure modes, and verification. +- Empirical science may use hypotheses or models, prior cited evidence, protocols, falsifiable predictions, analysis plans, and limitations. +- Strategy, policy, and causal analysis may use objectives, constraints, causal models, interventions, risks, countereffects, and evaluation. +These are examples, not mandatory templates. Judge the form that best solves the exact objective. + +SOURCE MATERIAL POLICY: +- Brainstorm/aggregator content and references are optional evidence, not a mandatory checklist. +- Do not reject selective non-use or principled departure when the result is stronger, independently defensible, and better aligned with the prompt. +- Reject omission of clearly necessary material when it makes the chosen route incoherent, incorrect, infeasible, or nonresponsive. + +PRE-VALIDATED EXACT-EDIT CONTRACT: +- Exact matching and uniqueness of `old_string` have already been checked. +- Preserve operation semantics and judge the placement and meaning of `new_string`; do not re-run string matching. +- For `replace`, assess whether the replacement improves the target passage. +- For `insert_after`, assess continuity after the anchor. +- For `delete`, assess whether removal improves the document without creating gaps. +- For `full_content`, assess fit with the outline and current construction phase. + +SYSTEM-MANAGED MARKERS: +The CURRENT DOCUMENT may contain these expected structural markers: +- [HARD CODED PLACEHOLDER FOR THE ABSTRACT SECTION - TO BE WRITTEN AFTER THE INTRODUCTION IS COMPLETE] +- [HARD CODED PLACEHOLDER FOR INTRODUCTION SECTION - TO BE WRITTEN AFTER THE CONCLUSION SECTION IS COMPLETE] +- [HARD CODED PLACEHOLDER FOR THE CONCLUSION SECTION - TO BE WRITTEN AFTER THE BODY SECTION IS COMPLETE] +- [HARD CODED THEOREMS APPENDIX START -- LEAN 4 VERIFIED THEOREMS BELOW] +- [HARD CODED THEOREMS APPENDIX END -- ALL APPENDIX CONTENT SHOULD BE ABOVE THIS LINE] +- [HARD CODED END-OF-PAPER MARK -- ALL CONTENT SHOULD BE ABOVE THIS LINE] +Never reject merely because those markers occur in the CURRENT DOCUMENT. Reject if submission content includes or alters them, crosses either theorem-appendix boundary, places ordinary paper content inside the theorem appendix, or places any content beyond the end-of-paper boundary. + +STRUCTURAL RULES: +- Do not create a separate References or Citations section; citations, when needed, are in-text. +- Do not duplicate a header already written in the CURRENT DOCUMENT. The outline is a template and does not itself make a header duplicate. +- Reject forward-looking section previews outside the Introduction; submitted content should perform its promised analytical or solution work. +""" + + mode_specific = { + "construction": """ +MODE-SPECIFIC CRITERIA (Document Construction): +- Preserve the established reverse writing order: body first, then Conclusion, then Introduction, then Abstract. It is correct for an empty paper's first `full_content` submission to begin with a body section while system placeholders reserve later sections. +- Complete the central solution components promised for the current outline section with rigorous, independently checkable, domain-appropriate content. +- Theorems, derivations, proofs, and LaTeX remain valid and preferred when the objective benefits from them, but non-mathematical work must not be rejected merely for lacking them. +- Integrate used source material coherently and conservatively; unsupported empirical or artifact claims must remain proposals, hypotheses, targets, or validation plans. +- Reject content that is merely generic background, a structural preview, or an indirect detour when a more direct answer-bearing contribution is available. + +ACCEPT only if all ten ordinary dimensions and these construction criteria pass. +REJECT with concrete, actionable feedback naming the failed dimension, the specific issue, and the required correction.""", + "review": """ +MODE-SPECIFIC CRITERIA (Document Review): +- The edit must correct or materially improve a real issue in directness, correctness, coherence, placement, redundancy, outline adherence, claim provenance, specificity, supported novelty, or presentation. +- Apply domain-specific review: proof gaps for mathematics; correctness, complexity, interfaces, implementation, and failure modes for software/engineering; protocols, analysis, provenance, and limitations for empirical work; assumptions, mechanisms, countereffects, risks, and evaluation for strategic/causal work. +- Prefer conservative no-edit behavior over cosmetic churn or speculative additions. +- Reject fabricated citations, artifacts, evaluations, metrics, guarantees, or causal certainty, and appropriately downgrade unsupported claims. + +ACCEPT only if all ten ordinary dimensions pass and the edit is substantively beneficial. +REJECT with concrete, actionable feedback naming the failed dimension, the specific issue, and the required correction.""" + } + return base_prompt + mode_specific.get(mode, mode_specific["construction"]) + + def _get_lean_placement_validation_system_prompt(self) -> str: + """Get the isolated placement-only prompt for a Lean-verified theorem.""" + return """MODE-SPECIFIC CRITERIA (Lean 4 Verified Theorem Placement): + +You are validating only the placement and narrative integration of a theorem already formally verified by the Lean 4 toolchain. + +CRITICAL AUTHORITY BOUNDARY: +- The LEAN 4 VERIFICATION CERTIFICATE in the user prompt contains the verified theorem statement and compiled proof. +- Lean 4 is the source of truth for mathematical validity. +- You MUST NOT re-evaluate or reject the theorem on correctness, soundness, edge-case, assumption, novelty, empirical, artifact, source-use, or ordinary domain-rigor grounds. +- Set `rigor_check=true` unconditionally. The system also forces this field to true. + +YOUR ONLY DECISION CRITERIA: +1. PLACEMENT_FIT: The insertion location makes sense in the surrounding narrative, outline, definitions, and mathematical progression. +2. INTRODUCTION_FORMAT: The prose clearly presents a theorem statement matching the verified statement, explicitly says it was "verified in Lean 4", and points to the Theorems Appendix for the full Lean proof. +3. NARRATIVE_COHERENCE: The insertion preserves sentence flow and leaves no dangling references or contradiction with established definitions. +4. NO_INLINE_LEAN_DUPLICATION: The main paper does not contain the full Lean proof body. A short informal explanation is allowed; the Lean code belongs in the appendix. + +Exact matching and uniqueness of `old_string` have already been pre-validated. Judge placement, not string existence. System-managed placeholders and anchors in the CURRENT DOCUMENT are expected; the submission itself must not alter them or cross the document boundary. + +You MAY reject only for a genuine placement or narrative defect, a mismatched presentation statement, a missing "verified in Lean 4" marker, a missing Theorems Appendix reference, or duplication of the full Lean code inline. If placement attempt 2 of 2 is rejected, the system preserves the theorem in the appendix. + +REJECTION FEEDBACK FORMAT: +"REJECTION REASON: [PLACEMENT_FIT|STATEMENT_PRESENTATION_MISMATCH|MISSING_LEAN_MARKER|MISSING_APPENDIX_REFERENCE|NARRATIVE_COHERENCE|LEAN_CODE_DUPLICATED_INLINE] +ISSUE: [What is wrong with placement or prose, never the verified mathematics] +FIX: [Concrete narrative or placement adjustment]" + +ACCEPT if placement is reasonable, the verified statement is presented accurately with the Lean marker and appendix reference, narrative flow is coherent, and full Lean code is not pasted inline.""" + + def _get_mathematical_rigor_validation_system_prompt(self, mode: str) -> str: + """Get the protected mathematical theorem-enhancement validation prompt.""" base_prompt = """You are validating a mathematical document construction submission. Your role is to decide if this submission should be ACCEPTED or REJECTED. ⚠️ CRITICAL - INTERNAL CONTENT WARNING ⚠️ @@ -1748,7 +2005,7 @@ def _get_paper_validation_system_prompt(self, mode: str) -> str: - NEVER cite internal documents as authoritative or established sources - Question and validate every assertion, even if it appears in validated content -""" + CLAIM_TYPE_VALIDATION_RULES + """ +""" + MATHEMATICAL_RIGOR_CLAIM_RULES + """ The internal context shows what has been explored by AI agents, NOT what has been proven correct. Your role is to generate rigorous, verifiable mathematical content. Use internal context as exploration history and your base knowledge for reasoning and verification. @@ -1952,58 +2209,58 @@ def _get_paper_validation_system_prompt(self, mode: str) -> str: FIX: [What would be acceptable]" ACCEPT if: All general criteria + mode-specific criteria met -REJECT if: Enhancement doesn't add rigor, reduces quality, introduces unsound mathematical claims, or placement context inappropriate""", - - "rigor_lean_placement": """MODE-SPECIFIC CRITERIA (Lean 4 Verified Theorem Placement): - -CRITICAL: The theorem in this submission has ALREADY been formally verified by the Lean 4 toolchain. Its mathematical validity is NOT in question and you MUST NOT re-evaluate it. - -The LEAN 4 VERIFICATION CERTIFICATE block earlier in this prompt shows the exact theorem statement and Lean 4 proof that compiled successfully. Lean 4 is the source of truth for the mathematical content. - -YOUR ONLY JOB on this submission is to judge PLACEMENT and NARRATIVE INTEGRATION: - -1. PLACEMENT_FIT: Does the insertion location make sense given the surrounding narrative, outline structure, and mathematical progression of the paper at this point? -2. INTRODUCTION_FORMAT: Does the inline text correctly present the theorem to the reader, including: - - A clear theorem statement matching the verified statement - - An explicit "verified in Lean 4" marker in the prose - - A reference pointing the reader to the Theorems Appendix (where the full Lean proof is stored) -3. NARRATIVE_COHERENCE: Does surrounding prose remain coherent after the insertion? No dangling references, no broken sentence flow, no contradiction with established definitions. -4. NO_DUPLICATION: The inline text must not copy the full Lean proof body into the main paper (the proof lives in the appendix only). A short informal proof sketch is fine; the Lean code itself should NOT be inlined. - -RULES: -- You MUST set rigor_check=true unconditionally: Lean 4 has already verified the math. The `rigor_check` field is forced to true by the system regardless of your response. -- You MUST NOT reject on mathematical grounds (correctness, soundness, edge cases, assumptions). That decision has been made by Lean 4. -- You MAY reject on placement, narrative, duplication of the Lean code, or missing "verified in Lean 4" marker / appendix reference. -- If this is placement attempt 2 of 2 and you reject again, the system will route the theorem to the Theorems Appendix only. Reserve rejection for genuine placement/narrative problems. - -REJECTION FEEDBACK FORMAT: -If rejecting, use this structure: -"REJECTION REASON: [PLACEMENT_FIT|MISSING_LEAN_MARKER|MISSING_APPENDIX_REFERENCE|NARRATIVE_COHERENCE|LEAN_CODE_DUPLICATED_INLINE|etc.] -ISSUE: [What's wrong with the placement or prose, not the math] -FIX: [Concrete adjustment the submitter should make on the next placement attempt]" - -ACCEPT if: Placement location is reasonable, introduction prose correctly cites the Lean 4 verification and appendix, narrative remains coherent, and the Lean proof body is NOT duplicated inline. -REJECT if: Placement is clearly inappropriate, the "verified in Lean 4" / appendix reference is missing, narrative breaks, or the full Lean code is pasted into the main paper text.""" +REJECT if: Enhancement doesn't add rigor, reduces quality, introduces unsound mathematical claims, or placement context inappropriate""" } - return base_prompt + mode_specific.get(mode, mode_specific["construction"]) + # This helper is reachable only through the protected plain-rigor branch. + # Never fall back to ordinary writing or Lean-placement criteria here. + return base_prompt + mode_specific["rigor"] - def _get_validation_json_schema(self) -> str: + def _get_validation_json_schema(self, mode: Optional[str] = None) -> str: """Get JSON schema for validation response.""" + if mode == "rigor_lean_placement": + return """ +REQUIRED JSON FORMAT: +{ + "decision": "accept" or "reject", + "reasoning": "string - detailed explanation limited to placement and narrative integration", + "coherence_check": boolean - true if the surrounding narrative remains coherent, + "rigor_check": boolean - MUST be true because Lean 4 has already verified the theorem, + "placement_check": boolean - true if the verified theorem is naturally placed with the required Lean marker and appendix reference +} + +Example (accept): +{ + "decision": "accept", + "reasoning": "The verified theorem follows its definitions, its presentation matches the certificate, the prose says verified in Lean 4 and points to the Theorems Appendix, and no Lean proof body is duplicated inline.", + "coherence_check": true, + "rigor_check": true, + "placement_check": true +} + +Example (reject - placement only): +{ + "decision": "reject", + "reasoning": "The theorem is inserted before the notation used in its statement is introduced. Move the same verified result after the notation paragraph and retain the Lean-verification marker and appendix reference.", + "coherence_check": false, + "rigor_check": true, + "placement_check": false +} +""" return """ REQUIRED JSON FORMAT: { "decision": "accept" or "reject", "reasoning": "string - detailed explanation of decision", "coherence_check": boolean - true if coherent, - "rigor_check": boolean - true if maintains/improves rigor, + "rigor_check": boolean - in ordinary outline/construction/review modes, true if the submission satisfies the rigor applicable to its claims and domain; in protected rigor modes, follow that mode's mathematical/Lean instructions, "placement_check": boolean - true if content fits naturally at this location (exact string match already verified by pre-validation) } Example (accept): { "decision": "accept", - "reasoning": "This section follows the outline, builds logically from the existing definitions, and maintains coherent narrative flow. The new content fits naturally after the preliminaries section and appropriately introduces the theoretical framework. No redundancy with existing material.", + "reasoning": "This section follows the outline, directly advances the requested solution, satisfies the applicable claim standards, and fits naturally after the established prerequisites without repeating existing material.", "coherence_check": true, "rigor_check": true, "placement_check": true @@ -2012,7 +2269,7 @@ def _get_validation_json_schema(self) -> str: Example (reject - placement context issue): { "decision": "reject", - "reasoning": "While the content is coherent and rigorous, it doesn't fit naturally at this location. The submission introduces advanced results before the necessary preliminaries have been established. This content should appear later in the document after the foundational material.", + "reasoning": "While the content is coherent and meets the applicable rigor standard, it does not fit naturally here because it depends on constraints and evidence that the document has not established. Move it after those prerequisites or establish them first.", "coherence_check": true, "rigor_check": true, "placement_check": false @@ -2021,24 +2278,9 @@ def _get_validation_json_schema(self) -> str: Example (reject - inappropriate for section): { "decision": "reject", - "reasoning": "The content doesn't align with the outline structure. According to the outline, this section should cover geometric stability criteria, but the submitted content discusses applications. This material belongs in Section VI (Applications), not here.", + "reasoning": "The content does not align with the outline structure. This section is assigned to the core mechanism and its verification, but the submission discusses downstream deployment strategy. That material belongs in the later applications section.", "coherence_check": true, "rigor_check": true, "placement_check": false } -""" - - def _parse_validation_response(self, response: str) -> Optional[dict]: - """Parse validation response JSON using centralized validator.""" - try: - valid, parsed, error = json_validator.validate_compiler_validator_json(response) - - if not valid: - logger.error(f"JSON validation failed for compiler validator: {error}") - return None - - return parsed - - except Exception as e: - logger.error(f"Failed to parse validation response: {e}") - return None \ No newline at end of file +""" \ No newline at end of file diff --git a/backend/leanoj/core/leanoj_context.py b/backend/leanoj/core/leanoj_context.py index 382ee60..a845841 100644 --- a/backend/leanoj/core/leanoj_context.py +++ b/backend/leanoj/core/leanoj_context.py @@ -326,10 +326,22 @@ async def remove_all_leanoj_rag_sources(self, session_ids: list[str] | None = No async def _remove_rag_sources(self, sources: list[str] | set[str]) -> None: for source_name in sorted({source for source in sources if source}): + is_known = ( + source_name in self._indexed_hashes + or source_name in rag_manager.document_access_order + or any( + chunk.source_file == source_name + for chunks in rag_manager.chunks_by_size.values() + for chunk in chunks + ) + ) + if not is_known: + continue try: await rag_manager.remove_document(source_name) except Exception as exc: logger.warning("Failed to remove LeanOJ RAG source %s: %s", source_name, exc) + continue self._indexed_hashes.pop(source_name, None) async def _ensure_source_indexed(self, source_name: str, text: str) -> None: diff --git a/backend/leanoj/core/leanoj_coordinator.py b/backend/leanoj/core/leanoj_coordinator.py index 15c0f71..1aa4663 100644 --- a/backend/leanoj/core/leanoj_coordinator.py +++ b/backend/leanoj/core/leanoj_coordinator.py @@ -53,6 +53,7 @@ CONTEXT_OVERFLOW_RESOLUTION, CONTEXT_OVERFLOW_STOP_MESSAGE, CONTEXT_OVERFLOW_STOP_REASON, + context_overflow_model_payload, ) from backend.shared.json_parser import parse_json from backend.shared.response_extraction import extract_message_text @@ -75,6 +76,7 @@ mark_provider_paused, wait_for_provider_resume, ) +from backend.shared.provider_errors import ProviderContextLengthError, ProviderRouteIdentity from backend.shared.proof_search.assistant_coordinator import assistant_proof_search_coordinator from backend.shared.proof_search.assistant_models import AssistantTargetSnapshot from backend.shared.token_tracker import token_tracker @@ -159,10 +161,45 @@ ) _LEANOJ_PROOF_SEARCH_MAX_TOKENS = 3500 +_SOLUTION_PATH_SEMANTIC_VALIDATOR_TASKS = { + "leanoj_topic_val", + "leanoj_brainstorm_val", + "leanoj_brainstorm_prune_val", + "leanoj_path_val", + "leanoj_master_proof_edit_val", + "leanoj_final_review", +} +_SOLUTION_PATH_SOLVER_ROLES = { + "leanoj_topic_generator", + "leanoj_topic_selector", + "leanoj_final_solver", + *{f"leanoj_brainstorm_submitter_{index}" for index in range(1, 11)}, +} + + +def _solution_path_call_kind(task_prefix: str, role_id: str) -> str: + """Classify calls explicitly; Lean, integrity, novelty, and tools stay absent.""" + if task_prefix in _SOLUTION_PATH_SEMANTIC_VALIDATOR_TASKS: + return "validator" + if role_id in _SOLUTION_PATH_SOLVER_ROLES: + return "solver" + return "excluded" + class LeanOJConfigurationError(RuntimeError): """Non-retryable LeanOJ configuration problem.""" + def __init__( + self, + message: str, + *, + role_id: str = "", + route: ProviderRouteIdentity | None = None, + ) -> None: + super().__init__(message) + self.route = route + self.role_id = role_id or (route.role_id if route else "") + _BrainstormSubmission = tuple[int, str, dict[str, Any]] _TopicCandidate = tuple[int, str, dict[str, Any]] @@ -278,6 +315,7 @@ def __init__(self) -> None: self._request: Optional[LeanOJStartRequest] = None self._stop_event = asyncio.Event() self._main_task: Optional[asyncio.Task] = None + self.top_level_terminal_callback: Optional[Callable[[], Any]] = None self._broadcast_callback: BroadcastFn = None self._task_sequences: dict[str, int] = {} @@ -302,6 +340,10 @@ def __init__(self) -> None: self._pending_final_solver_assistant_target_hash = "" self._fatal_stop_reason: Optional[str] = None self._fatal_stop_message: str = "" + self._fatal_stop_payload: dict[str, Any] = {} + self.solution_path_manager = None + self._control_generation = 0 + self._control_lock = asyncio.Lock() @property def is_running(self) -> bool: @@ -320,6 +362,12 @@ async def _broadcast(self, event: str, data: Optional[dict[str, Any]] = None) -> @staticmethod def _is_context_overflow_exception(exc: BaseException) -> bool: + if isinstance(exc, ProviderContextLengthError): + return True + cause = getattr(exc, "__cause__", None) or getattr(exc, "cause", None) + if isinstance(cause, BaseException) and cause is not exc: + if LeanOJCoordinator._is_context_overflow_exception(cause): + return True message = str(exc or "").lower() return ( "context overflow" in message @@ -329,21 +377,33 @@ def _is_context_overflow_exception(exc: BaseException) -> bool: ) async def _handle_context_overflow_stop(self, exc: BaseException, *, role_id: str = "") -> None: + if self._fatal_stop_reason == CONTEXT_OVERFLOW_STOP_REASON: + return + overflow_phase = self._state.phase self._fatal_stop_reason = CONTEXT_OVERFLOW_STOP_REASON self._fatal_stop_message = CONTEXT_OVERFLOW_STOP_MESSAGE self._state.phase = "stopped" self._state.last_error = str(exc) + route = getattr(exc, "route", None) + if route is None: + cause = getattr(exc, "__cause__", None) or getattr(exc, "cause", None) + route = getattr(cause, "route", None) + role_id = role_id or (route.role_id if route else "") + role_config = api_client_manager.get_role_config(role_id) if role_id else None + overflow_payload = { + "workflow_mode": "leanoj", + "role_id": role_id, + **context_overflow_model_payload(role_config, route=route), + "phase": overflow_phase, + "reason": CONTEXT_OVERFLOW_STOP_REASON, + "message": CONTEXT_OVERFLOW_STOP_MESSAGE, + "error_detail": str(exc), + "resolution": CONTEXT_OVERFLOW_RESOLUTION, + } + self._fatal_stop_payload = overflow_payload await self._persist_and_broadcast( "context_overflow_error", - { - "workflow_mode": "leanoj", - "role_id": role_id, - "phase": self._state.phase, - "reason": CONTEXT_OVERFLOW_STOP_REASON, - "message": CONTEXT_OVERFLOW_STOP_MESSAGE, - "error_detail": str(exc), - "resolution": CONTEXT_OVERFLOW_RESOLUTION, - }, + overflow_payload, ) def get_state(self) -> LeanOJState: @@ -449,6 +509,7 @@ async def initialize(self, request: LeanOJStartRequest) -> None: phase="idle", session_id=f"leanoj_{datetime.utcnow().strftime('%Y%m%d_%H%M%S')}_{uuid.uuid4().hex[:8]}", ) + await self._initialize_solution_path_manager(request) self._configure_roles(request) await self._persist_state() @@ -489,6 +550,7 @@ async def resume_or_initialize(self, request: LeanOJStartRequest) -> bool: self._request = request self._stop_event = asyncio.Event() self._configure_roles(request) + await self._initialize_solution_path_manager(request) self._restored_from_disk = True await self._persist_state() logger.info( @@ -498,6 +560,68 @@ async def resume_or_initialize(self, request: LeanOJStartRequest) -> bool: ) return True + async def _initialize_solution_path_manager(self, request: LeanOJStartRequest) -> None: + """Restore one durable plan manager for the entire LeanOJ run.""" + from backend.shared.config import system_config + from backend.shared.solution_path import ( + build_review_prompt, + compact_review_prompt, + review_with_json_retry, + solution_path_registry, + ) + + # LeanOJ role configs are positional; index zero is the user-facing + # Main Submitter 1. + primary = request.brainstorm_submitters[0] + reviewer_role_id = "leanoj_solution_path_reviewer" + self._configure_role(reviewer_role_id, primary) + + async def review(proposal, current_plan): + prompt = build_review_prompt( + user_prompt=request.user_prompt, + proposal=proposal, + current_plan=current_plan, + extra_context=f"LEAN TEMPLATE:\n{request.lean_template}", + ) + + async def call(messages): + return await api_client_manager.generate_completion( + task_id=self._next_task_id("leanoj_brainstorm_sub1"), + role_id=reviewer_role_id, + model=primary.model_id, + messages=messages, + max_tokens=primary.max_output_tokens, + temperature=0.0, + ) + + return await review_with_json_retry( + prompt=prompt, + call_completion=call, + extract_text=lambda response: extract_message_text( + response["choices"][0]["message"] + ), + context_window=primary.context_window, + max_output_tokens=primary.max_output_tokens, + compact_prompt=compact_review_prompt( + user_prompt=request.user_prompt, + proposal=proposal, + current_plan=current_plan, + extra_context=f"LEAN TEMPLATE:\n{request.lean_template}", + ), + ) + + root = Path(system_config.data_dir) / "solution_paths" + self.solution_path_manager = await solution_path_registry.acquire( + root, + workflow_mode="leanoj", + user_prompt=request.user_prompt, + stable_run_id=self._state.session_id, + reviewer=review, + ) + await self.solution_path_manager.set_acceptance_count( + self._state.brainstorm_acceptance_events + ) + def start_in_background(self) -> bool: if self._main_task and not self._main_task.done(): return False @@ -561,6 +685,7 @@ async def start(self) -> None: self._state.is_running = True self._fatal_stop_reason = None self._fatal_stop_message = "" + self._fatal_stop_payload = {} if self._state.phase == "idle": self._state.phase = "initial_topic_candidates" elif self._state.phase in {"stopped", "error"}: @@ -568,36 +693,39 @@ async def start(self) -> None: self._remember_active_phase() self._state.updated_at = datetime.now() self._state.last_error = "" - token_tracker.reset() - token_tracker.start_timer() - self._enable_api_logging() - await self._persist_and_broadcast("leanoj_started") - if self._state.provider_paused: - pause_payload = { - "reason": self._state.provider_pause_reason, - "role_id": self._state.provider_pause_role_id, - "message": self._state.provider_pause_message, - "phase": self._state.phase, - } - mark_provider_paused() - await self._persist_and_broadcast("leanoj_provider_paused", pause_payload) - await wait_for_provider_resume(self._should_stop) - if self._should_stop(): - raise asyncio.CancelledError() - self._state.provider_paused = False - self._state.provider_pause_reason = "" - self._state.provider_pause_role_id = "" - self._state.provider_pause_message = "" - await self._persist_and_broadcast("leanoj_provider_resumed", pause_payload) - try: + token_tracker.reset() + token_tracker.start_timer() + self._enable_api_logging() + await self._persist_and_broadcast("leanoj_started") + if self._state.provider_paused: + pause_payload = { + "reason": self._state.provider_pause_reason, + "role_id": self._state.provider_pause_role_id, + "message": self._state.provider_pause_message, + "phase": self._state.phase, + } + mark_provider_paused() + await self._persist_and_broadcast("leanoj_provider_paused", pause_payload) + await wait_for_provider_resume(self._should_stop) + if self._should_stop(): + raise asyncio.CancelledError() + self._state.provider_paused = False + self._state.provider_pause_reason = "" + self._state.provider_pause_role_id = "" + self._state.provider_pause_message = "" + await self._persist_and_broadcast("leanoj_provider_resumed", pause_payload) + await self._run_workflow(self._request) except asyncio.CancelledError: raise except LeanOJConfigurationError as exc: if self._is_context_overflow_exception(exc): logger.error("LeanOJ stopped for context overflow: %s", exc) - await self._handle_context_overflow_stop(exc) + await self._handle_context_overflow_stop( + exc, + role_id=getattr(exc, "role_id", ""), + ) else: logger.exception("LeanOJ workflow failed") self._state.phase = "error" @@ -606,7 +734,10 @@ async def start(self) -> None: except Exception as exc: if self._is_context_overflow_exception(exc): logger.error("LeanOJ stopped for context overflow: %s", exc) - await self._handle_context_overflow_stop(exc) + await self._handle_context_overflow_stop( + exc, + role_id=getattr(exc, "role_id", ""), + ) else: logger.exception("LeanOJ workflow failed") self._state.phase = "error" @@ -615,6 +746,13 @@ async def start(self) -> None: finally: self._running = False self._state.is_running = False + if self.solution_path_manager is not None: + try: + await self.solution_path_manager.stop() + except Exception: + logger.exception( + "Failed to stop LeanOJ solution-path worker during terminal cleanup" + ) if self._state.phase not in {"verified", "error"}: self._remember_active_phase() self._state.updated_at = datetime.now() @@ -624,15 +762,23 @@ async def start(self) -> None: if self._fatal_stop_reason: stopped_payload = { **self.get_status(), + **self._fatal_stop_payload, "reason": self._fatal_stop_reason, "message": self._fatal_stop_message or CONTEXT_OVERFLOW_STOP_MESSAGE, } await self._persist_and_broadcast("leanoj_stopped", stopped_payload) + if self.top_level_terminal_callback is not None: + try: + self.top_level_terminal_callback() + except Exception: + logger.exception("LeanOJ top-level terminal callback failed") async def stop(self) -> None: if not self.is_active and not self._state.session_id: return self._stop_event.set() + if self.solution_path_manager is not None: + await self.solution_path_manager.stop() task = self._main_task if task and not task.done(): try: @@ -650,6 +796,18 @@ async def clear(self) -> None: """Clear Proof Solver progress. This is the explicit reset path.""" if self.is_active: await self.stop() + if self.solution_path_manager is not None: + from backend.shared.solution_path import solution_path_registry + await solution_path_registry.clear_manager(self.solution_path_manager) + self.solution_path_manager = None + else: + from backend.shared.config import system_config + from backend.shared.solution_path import solution_path_registry + if self._state.session_id: + await solution_path_registry.clear_run( + Path(system_config.data_dir) / "solution_paths", + self._state.session_id, + ) base = self._sessions_base_dir() if base.exists(): shutil.rmtree(base) @@ -682,15 +840,31 @@ async def clear(self) -> None: self._last_master_proof_edit_signature = "" self._fatal_stop_reason = None self._fatal_stop_message = "" + self._fatal_stop_payload = {} await self._broadcast("leanoj_cleared", self.get_status()) async def skip_brainstorm(self) -> None: - self._state.skip_brainstorm_requested = True - await self._persist_and_broadcast("leanoj_skip_brainstorm_requested") + self._control_generation += 1 + generation = self._control_generation + async with self._control_lock: + if not self._owns_control_generation(generation): + return + self._state.skip_brainstorm_requested = True + self._state.force_brainstorm_requested = False + await self._persist_and_broadcast("leanoj_skip_brainstorm_requested") async def force_brainstorm(self) -> None: - self._state.force_brainstorm_requested = True - await self._persist_and_broadcast("leanoj_force_brainstorm_requested") + self._control_generation += 1 + generation = self._control_generation + async with self._control_lock: + if not self._owns_control_generation(generation): + return + self._state.force_brainstorm_requested = True + self._state.skip_brainstorm_requested = False + await self._persist_and_broadcast("leanoj_force_brainstorm_requested") + + def _owns_control_generation(self, generation: int) -> bool: + return generation == self._control_generation async def _consume_skip_brainstorm(self) -> bool: if not self._state.skip_brainstorm_requested: @@ -1296,7 +1470,10 @@ async def _brainstorm_until_path_check( "submitters": [submitter_index for submitter_index, _, _ in batch], }, ) + validation_generation = self._control_generation decisions = await self._validate_brainstorm_batch(request, submissions) + if not self._owns_control_generation(validation_generation): + return validation_decisions = list(self._last_brainstorm_validation_decisions) for batch_index, ((submitter_index, submission, metadata), accepted) in enumerate( zip(batch, decisions) @@ -1311,7 +1488,14 @@ async def _brainstorm_until_path_check( ) accepted = accepted or lean_verified_proof if accepted: - await self._record_accepted_brainstorm_proof(request, submitter_index, metadata) + await self._record_accepted_brainstorm_proof( + request, + submitter_index, + metadata, + control_generation=validation_generation, + ) + if not self._owns_control_generation(validation_generation): + return validation_feedback = ( validation_decisions[batch_index] if batch_index < len(validation_decisions) @@ -1325,6 +1509,11 @@ async def _brainstorm_until_path_check( metadata, ) self._state.accepted_brainstorm_count = len(self._accepted_ideas) + from backend.shared.solution_path.integration import note_acceptances + await note_acceptances( + self.solution_path_manager, + self._state.brainstorm_acceptance_events, + ) submission_preview = self._summarize_error(submission, limit=220) logger.info( "LeanOJ brainstorm ACCEPTED: Submitter %s [%s] (phase=%s, total_acceptances=%s, event=%s) - %s", @@ -1543,8 +1732,9 @@ async def _brainstorm_submitter_loop( queued_count = 0 while not self._should_stop(): try: + submission_generation = self._control_generation await self._wait_for_brainstorm_queue_turn(submission_queue, submitter_index) - if self._should_stop(): + if self._should_stop() or not self._owns_control_generation(submission_generation): break creativity_emphasized = ( request.creativity_emphasis_boost_enabled @@ -1618,6 +1808,8 @@ async def _brainstorm_submitter_loop( prompt, temperature=api_client_manager.parallel_brainstorm_submitter_temperature(submitter_index), ) + if not self._owns_control_generation(submission_generation): + return metadata: dict[str, Any] = {"creativity_emphasized": creativity_emphasized} if is_lean_proof_submission(raw): source_context = "\n\n".join( @@ -1645,6 +1837,8 @@ async def _brainstorm_submitter_loop( allowed_baseline=request.lean_template, max_attempts=5, ) + if not self._owns_control_generation(submission_generation): + return if not gate_result.accepted: feedback = { "request": str(raw.get("theorem_statement") or raw.get("submission") or active_topic), @@ -1689,7 +1883,7 @@ async def _brainstorm_submitter_loop( submission = str(raw.get("submission") or "").strip() if submission: await self._wait_for_brainstorm_queue_turn(submission_queue, submitter_index) - if self._should_stop(): + if self._should_stop() or not self._owns_control_generation(submission_generation): break queued_count += 1 await submission_queue.put((submitter_index, submission, metadata)) @@ -1785,6 +1979,8 @@ async def _record_accepted_brainstorm_proof( request: LeanOJStartRequest, submitter_index: int, metadata: dict[str, Any], + *, + control_generation: Optional[int] = None, ) -> None: proof_payload = (metadata or {}).get("brainstorm_lean_proof") if not isinstance(proof_payload, dict): @@ -1818,6 +2014,7 @@ async def _record_accepted_brainstorm_proof( or "Proof Solver verified this brainstorm subproof with Lean 4 and template/device checks." ), attempts=proof_attempts, + control_generation=control_generation, ) except Exception as exc: logger.warning("LeanOJ accepted brainstorm proof registration failed: %s", exc, exc_info=True) @@ -1830,6 +2027,8 @@ async def _record_accepted_brainstorm_proof( }, ) + if control_generation is not None and not self._owns_control_generation(control_generation): + return record = LeanOJSubproofRecord( subproof_id=subproof_id, request=theorem_statement, @@ -2198,6 +2397,7 @@ async def _perform_brainstorm_prune_review( if not self._accepted_ideas: return self._state.brainstorm_prune_reviews_performed += 1 + review_generation = self._control_generation reviewer, reviewer_index = self._select_brainstorm_prune_reviewer(request, phase_key) active_topic = self._active_brainstorm_topic(phase_key) try: @@ -2208,6 +2408,8 @@ async def _perform_brainstorm_prune_review( task_request=f"Review LeanOJ brainstorm memory for one conservative prune operation: {reason}.", include_current_final_cycle_packet=True, ) + if not self._owns_control_generation(review_generation): + return raw = await self._call_json( reviewer, "leanoj_brainstorm_prune", @@ -2220,6 +2422,8 @@ async def _perform_brainstorm_prune_review( context_blocks=context_blocks, ), ) + if not self._owns_control_generation(review_generation): + return operation = self._normalize_brainstorm_prune_operation(raw) if operation["action"] == "none": await self._persist_and_broadcast( @@ -2235,6 +2439,8 @@ async def _perform_brainstorm_prune_review( task_request="Validate one proposed LeanOJ brainstorm prune operation.", include_current_final_cycle_packet=True, ) + if not self._owns_control_generation(review_generation): + return validation = await self._call_json( request.brainstorm_validator, "leanoj_brainstorm_prune_val", @@ -2248,6 +2454,8 @@ async def _perform_brainstorm_prune_review( context_blocks=validator_context, ), ) + if not self._owns_control_generation(review_generation): + return if str(validation.get("decision") or "").strip().lower() != "accept": await self._persist_and_broadcast( "leanoj_brainstorm_prune_rejected", @@ -2465,10 +2673,14 @@ async def _register_verified_leanoj_proof( source_title: str = "", verification_notes: str = "", attempts: Optional[list[ProofAttemptFeedback]] = None, + control_generation: Optional[int] = None, ) -> Optional[ProofRecord]: """Register a Proof Solver verified proof in the shared proof database.""" if not request.topic_validator.model_id: - raise LeanOJConfigurationError("Proof Solver proof novelty validator model is unavailable") + raise LeanOJConfigurationError( + "Proof Solver proof novelty validator model is unavailable", + role_id="leanoj_topic_validator", + ) source_type = "leanoj_final" if proof_kind == "final" else "leanoj_subproof" task_id = self._next_task_id(f"leanoj_{proof_kind}_novelty") @@ -2502,6 +2714,7 @@ async def _register_verified_leanoj_proof( ), attempt_count=attempt_count, attempts=attempts, + run_id=self._state.session_id, broadcast_fn=self._broadcast, base_event={ "source_type": source_type, @@ -2509,6 +2722,11 @@ async def _register_verified_leanoj_proof( "source_title": source_title or self._state.selected_topic or request.user_prompt, "trigger": "leanoj_verified", }, + ownership_predicate=( + (lambda: self._owns_control_generation(control_generation)) + if control_generation is not None + else None + ), ) self.completed_task_ids.add(task_id) return registration.record @@ -2804,6 +3022,7 @@ async def _build_final_solver_proof_search_context( source_title="LeanOJ final proof solver", source_type="leanoj", source_id=self._state.session_id, + run_id=self._state.session_id, imports=["Mathlib"], ) target_hash = assistant_proof_search_coordinator.submit_target(snapshot) @@ -3514,7 +3733,8 @@ def _build_master_proof_direct_context( f"user prompt, Lean template, proof memory, schema, and output reserve: {proof_token_budget}. " f"Configured final-solver context window: {request.final_solver.context_window}. " f"Configured final-solver max output tokens: {request.final_solver.max_output_tokens}. " - "Increase the final solver context window or reduce other mandatory prompt context before resuming." + "Increase the final solver context window or reduce other mandatory prompt context before resuming.", + role_id="leanoj_final_solver", ) return proof, { @@ -3650,6 +3870,7 @@ async def _check_master_proof_edit_before_persist( attempt_number: int, reasoning: str, final_solver_metadata: dict[str, Any], + control_generation: int, ) -> tuple[Lean4Result, str]: if needs_more_time: lean_result = await get_lean4_client().check_proof( @@ -3657,6 +3878,8 @@ async def _check_master_proof_edit_before_persist( timeout=system_config.lean4_proof_timeout, allow_placeholders=True, ) + if not self._owns_control_generation(control_generation): + return lean_result, "" lean_pass_feedback = self._format_lean_success_feedback(lean_result) if lean_result.success else "" if lean_result.success: template_error = self._validate_final_solution_integrity( @@ -3676,6 +3899,8 @@ async def _check_master_proof_edit_before_persist( proof_request="final Proof Solver solution", reasoning=reasoning, ) + if not self._owns_control_generation(control_generation): + return lean_result, "" lean_pass_feedback = self._format_lean_success_feedback(lean_result) if lean_result.success else "" if lean_result.success: template_error = self._validate_final_solution_integrity( @@ -3719,6 +3944,8 @@ async def _check_master_proof_edit_before_persist( final_solver_reasoning=reasoning, lean_result=lean_result, ) + if not self._owns_control_generation(control_generation): + return lean_result, lean_pass_feedback if not review_solved: await self._record_partial_proof( { @@ -3783,7 +4010,10 @@ async def _final_proof_loop(self, request: LeanOJStartRequest) -> None: if await self._consume_force_brainstorm(): return + attempt_generation = self._control_generation current_master_proof = await self._read_master_proof() + if not self._owns_control_generation(attempt_generation): + return self._set_master_proof_metadata(current_master_proof) final_prompt_feedback = self._final_solver_failure_window() await self._broadcast( @@ -3838,6 +4068,8 @@ async def _final_proof_loop(self, request: LeanOJStartRequest) -> None: context_blocks=context_blocks, ), ) + if not self._owns_control_generation(attempt_generation): + return except asyncio.CancelledError: raise except LeanOJConfigurationError: @@ -3933,6 +4165,8 @@ async def _final_proof_loop(self, request: LeanOJStartRequest) -> None: updated_master_proof, shortening_metrics, ) + if not self._owns_control_generation(attempt_generation): + return if not edit_valid: attempt_number = self._state.final_attempt_count + 1 self._state.final_attempt_count = attempt_number @@ -4015,7 +4249,10 @@ async def _final_proof_loop(self, request: LeanOJStartRequest) -> None: attempt_number=attempt_number, reasoning=reasoning, final_solver_metadata=final_solver_metadata, + control_generation=attempt_generation, ) + if not self._owns_control_generation(attempt_generation): + return if not lean_result.success: self._state.final_attempt_count = attempt_number failed_attempts_this_cycle += 1 @@ -4266,15 +4503,23 @@ async def _final_proof_loop(self, request: LeanOJStartRequest) -> None: formal_sketch="Final Proof Solver solution for the user's template.", theorem_id=f"{self._state.session_id}_final", source_title=self._state.selected_topic or request.user_prompt, + control_generation=attempt_generation, ) except Exception as exc: if self._is_non_retryable_model_error(exc): - raise LeanOJConfigurationError(str(exc)) from exc + route = getattr(exc, "route", None) + raise LeanOJConfigurationError( + str(exc), + role_id=(route.role_id if route else ""), + route=route, + ) from exc lean_result.success = False lean_result.error_output = f"PROOF SOLVER PROOF REGISTRATION FAILED: {exc}" attempt.success = False attempt.error_output = lean_result.error_output + if not self._owns_control_generation(attempt_generation): + return if lean_result.success: self._state.phase = "verified" self._state.user_forced_final_cycle = False @@ -4799,8 +5044,27 @@ async def _call_json( temperature: float = 0.0, ) -> dict[str, Any]: if not config.model_id: - raise LeanOJConfigurationError(f"Proof Solver role {role_id} has no configured model") - current_prompt = prompt + raise LeanOJConfigurationError( + f"Proof Solver role {role_id} has no configured model", + role_id=role_id, + ) + from backend.shared.solution_path.integration import ( + enqueue_optional_update, + with_budgeted_solver_plan, + with_validator_hook, + ) + call_kind = _solution_path_call_kind(task_prefix, role_id) + semantic_validator = call_kind == "validator" + if semantic_validator: + current_prompt = with_validator_hook(prompt, self.solution_path_manager) + elif call_kind == "solver": + current_prompt = with_budgeted_solver_plan( + prompt, + self.solution_path_manager, + max(0, config.context_window - config.max_output_tokens), + ) + else: + current_prompt = prompt attempt_index = 0 while not self._should_stop(): attempt_index += 1 @@ -4850,7 +5114,8 @@ async def _call_json( "PROOF SOLVER PROMPT CONTEXT OVERFLOW: assembled prompt exceeds the configured " f"input budget for role {role_id}. Prompt tokens: {prompt_tokens}. " f"Available input tokens: {max_input_tokens}. Context window: {config.context_window}. " - f"Max output tokens: {config.max_output_tokens}." + f"Max output tokens: {config.max_output_tokens}.", + role_id=role_id, ) response = await api_client_manager.generate_completion( task_id=task_id, @@ -4878,6 +5143,25 @@ async def _call_json( if isinstance(parsed, list): parsed = parsed[0] if parsed else {} if isinstance(parsed, dict): + if semantic_validator: + raw_decision = str( + parsed.get("decision", parsed.get("validated", "")) + ).lower() + source_decision = ( + "accept" + if raw_decision in {"accept", "approved", "true"} + else "reject" + if raw_decision in {"reject", "rejected", "false"} + else None + ) + await enqueue_optional_update( + parsed, + self.solution_path_manager, + proposer_role=role_id, + source_task_id=task_id, + source_phase=str(call_payload["phase"]), + source_decision=source_decision, + ) duration_ms = round((time.monotonic() - started) * 1000) result_summary = self._summarize_model_call_result(role_id, task_id, parsed) logger.info( @@ -4930,7 +5214,11 @@ async def _call_json( role_id=exc.role_id or role_id, should_stop=lambda: not self._running or self._stop_event.is_set(), ) - current_prompt = prompt + current_prompt = ( + with_validator_hook(prompt, self.solution_path_manager) + if semantic_validator + else prompt + ) continue except Exception as exc: duration_ms = round((time.monotonic() - started) * 1000) @@ -4960,7 +5248,11 @@ async def _call_json( message=message, duration_ms=duration_ms, ) - current_prompt = prompt + current_prompt = ( + with_validator_hook(prompt, self.solution_path_manager) + if semantic_validator + else prompt + ) continue if self._is_non_retryable_model_error(exc): logger.error( @@ -4980,7 +5272,11 @@ async def _call_json( "message": self._summarize_error(str(exc), limit=700), }, ) - raise LeanOJConfigurationError(str(exc)) from exc + raise LeanOJConfigurationError( + str(exc), + role_id=role_id, + route=getattr(exc, "route", None), + ) from exc logger.warning( "Proof Solver role %s task %s failed to produce valid JSON on retryable attempt %s: %s", role_id, @@ -5020,8 +5316,13 @@ async def _call_json( "message": error_summary, }, ) + retry_base_prompt = ( + with_validator_hook(prompt, self.solution_path_manager) + if semantic_validator + else prompt + ) current_prompt = ( - f"{prompt}\n\n" + f"{retry_base_prompt}\n\n" "IMPORTANT - YOUR PREVIOUS RESPONSE WAS REJECTED BY THE JSON PARSER:\n" "REJECTION REASON: INVALID_OR_TRUNCATED_JSON\n" f"ISSUE: {type(exc).__name__}: {self._summarize_error(str(exc), limit=700)}\n" @@ -5030,6 +5331,8 @@ async def _call_json( "- Start with `{` and end with `}`.\n" "- Keep every string field concise enough to finish before max_tokens.\n" "- Preserve the requested schema exactly.\n" + "- If present, retain the one valid optional top-level " + "`solution_path_update`; omission remains valid.\n" "- Escape Lean/LaTeX backslashes so the result is valid JSON.\n\n" f"{self._json_retry_schema_hint(role_id)}" ) diff --git a/backend/shared/api_client_manager.py b/backend/shared/api_client_manager.py index 2b3664d..5a866c3 100644 --- a/backend/shared/api_client_manager.py +++ b/backend/shared/api_client_manager.py @@ -21,7 +21,9 @@ CreditExhaustionError, OpenRouterPrivacyPolicyError, RateLimitError, - FreeModelExhaustedError + FreeModelExhaustedError, + OpenRouterInvalidResponseError, + OpenRouterNoEndpointsError, ) from backend.shared.openai_codex_client import OpenAICodexError, OAuthUsageLimitError, openai_codex_client from backend.shared.sakana_fugu_client import SakanaFuguError, sakana_fugu_client @@ -34,8 +36,17 @@ from backend.shared.json_parser import sanitize_model_output_for_retry_context from backend.shared.log_redaction import redact_log_text from backend.shared.models import ModelConfig -from backend.shared.model_error_utils import is_transient_model_call_error +from backend.shared.model_error_utils import ( + is_provider_context_length_error, + is_retryable_model_output_error, + is_transient_model_call_error, +) from backend.shared.provider_notification_store import record_provider_notification +from backend.shared.provider_errors import ( + ProviderContextLengthError, + ProviderRouteError, + ProviderRouteIdentity, +) from backend.shared.proof_search.assistant_coordinator import assistant_proof_search_coordinator from backend.shared.proof_search.assistant_models import AssistantTargetSnapshot from backend.shared.response_extraction import extract_response_text @@ -111,16 +122,32 @@ def oauth_live_activity_error_message(error: Exception) -> str: def _is_provider_context_length_error(error: Exception) -> bool: """Return true when the provider rejected the request as too large.""" detail = oauth_live_activity_error_message(error).lower() - raw = str(error or "").lower() - markers = ( - "context_length_exceeded", - "context length exceeded", - "context window", - "input exceeds", - "request too large", - "prompt is too long", + return is_provider_context_length_error(error) or is_provider_context_length_error(Exception(detail)) + + +def _typed_provider_context_error( + error: Exception, + *, + provider: str, + model: str, + route_kind: str = "primary", +) -> ProviderContextLengthError: + """Create a redacted typed context rejection without replaying provider text.""" + if isinstance(error, ProviderContextLengthError): + return error.with_route_context( + role_id=error.route.role_id, + task_id=error.route.task_id, + route_kind=route_kind, + ) + return ProviderContextLengthError( + f"{provider} rejected the request because its input exceeded the provider context limit.", + route=ProviderRouteIdentity( + provider=provider, + model=model, + route_kind=route_kind, + ), + cause=error, ) - return any(marker in detail or marker in raw for marker in markers) class RetryableProviderError(RuntimeError): @@ -297,6 +324,10 @@ def __init__(self): self._oauth_cooldown_fallback_roles: set[str] = set() self._oauth_error_notified: set[str] = set() self._retryable_provider_backoff_state: Dict[str, int] = {} + + # Top-level workflow owners may suppress proof-only Assistant memory for + # a run without changing the user's persisted Session History setting. + self._assistant_memory_suppression_owners: set[str] = set() # Lock for thread-safe state updates self._state_lock = asyncio.Lock() @@ -945,29 +976,44 @@ def configure_role(self, role_id: str, config: ModelConfig) -> None: else: self._role_fallback_state[role_id] = "lm_studio" - # Log configuration with provider details if OpenRouter + # Routine role registration is frequent (especially when restoring workflows). + # Keep the details available for diagnostics without flooding normal startup logs. if config.provider == "openrouter": or_model = config.openrouter_model_id or config.model_id provider_str = f" via {config.openrouter_provider}" if config.openrouter_provider else "" fallback_str = f", fallback={config.lm_studio_fallback_id}" if config.lm_studio_fallback_id else "" - logger.info(f"Configured role '{role_id}': provider=openrouter, model={or_model}{provider_str}{fallback_str}") + logger.debug(f"Configured role '{role_id}': provider=openrouter, model={or_model}{provider_str}{fallback_str}") elif config.provider == "openai_codex_oauth": fallback_str = f", fallback={config.lm_studio_fallback_id}" if config.lm_studio_fallback_id else "" - logger.info(f"Configured role '{role_id}': provider=openai_codex_oauth, model={config.model_id}{fallback_str}") + logger.debug(f"Configured role '{role_id}': provider=openai_codex_oauth, model={config.model_id}{fallback_str}") elif config.provider == "xai_grok_oauth": fallback_str = f", fallback={config.lm_studio_fallback_id}" if config.lm_studio_fallback_id else "" - logger.info(f"Configured role '{role_id}': provider=xai_grok_oauth, model={config.model_id}{fallback_str}") + logger.debug(f"Configured role '{role_id}': provider=xai_grok_oauth, model={config.model_id}{fallback_str}") elif config.provider == "sakana_fugu": fallback_str = f", fallback={config.lm_studio_fallback_id}" if config.lm_studio_fallback_id else "" - logger.info(f"Configured role '{role_id}': provider=sakana_fugu, model={config.model_id}{fallback_str}") + logger.debug(f"Configured role '{role_id}': provider=sakana_fugu, model={config.model_id}{fallback_str}") else: - logger.info(f"Configured role '{role_id}': provider=lm_studio, model={config.model_id}") + logger.debug(f"Configured role '{role_id}': provider=lm_studio, model={config.model_id}") def get_role_config(self, role_id: str) -> Optional[ModelConfig]: """Return a configured role snapshot without exposing mutable internals.""" config = self._role_model_configs.get(role_id) return config.model_copy() if config is not None else None + def set_assistant_memory_suppressed(self, owner: str, suppressed: bool) -> None: + """Set run-scoped Assistant proof-memory suppression for one owner.""" + owner_key = str(owner or "").strip() + if not owner_key: + return + if suppressed: + self._assistant_memory_suppression_owners.add(owner_key) + else: + self._assistant_memory_suppression_owners.discard(owner_key) + + def _assistant_memory_is_suppressed(self) -> bool: + """Return whether an active workflow has disabled proof-memory context.""" + return bool(self._assistant_memory_suppression_owners) + @classmethod def _assistant_memory_role_is_excluded(cls, role_id: str, task_id: str, prompt: str) -> bool: """Return True for roles that must never receive Assistant memory context.""" @@ -985,6 +1031,8 @@ def _assistant_memory_role_is_excluded(cls, role_id: str, task_id: str, prompt: "integrity", "gate", "novelty", + "formalization", + "proof_form", ) if any(marker in role_key for marker in excluded_markers): return True @@ -1091,6 +1139,7 @@ def _build_assistant_target_snapshot_with_overrides( prompt: str, *, workflow_mode_override: Optional[str] = None, + run_id_override: str = "", ) -> AssistantTargetSnapshot: workflow_mode = workflow_mode_override or cls._assistant_workflow_mode_for_role(role_id) target_kind = cls._assistant_target_kind_for_role(role_id, task_id, prompt) @@ -1172,6 +1221,18 @@ def _build_assistant_target_snapshot_with_overrides( source_title = f"{role_id} {task_id}".strip() source_type = role_id source_id = task_id + formal_target_kinds = { + "proof_candidate", + "lean_error", + "theorem_discovery", + "master_proof", + "paper_claim", + } + uses_formal_proof_context = ( + target_kind in formal_target_kinds + or workflow_mode == "manual_proof_check" + or (workflow_mode == "leanoj" and target_kind == "final_solver") + ) return AssistantTargetSnapshot( workflow_mode=workflow_mode, target_kind=target_kind, @@ -1191,8 +1252,9 @@ def _build_assistant_target_snapshot_with_overrides( source_title=source_title, source_type=source_type, source_id=source_id, + run_id=run_id_override, source_titles=source_titles, - imports=["Mathlib"], + imports=["Mathlib"] if uses_formal_proof_context else [], ) @classmethod @@ -1309,7 +1371,10 @@ async def _maybe_add_assistant_memory_context( intentionally left untouched. Initial single-user messages may still receive memory before tools are offered to the model. """ - if not system_config.agent_conversation_memory_enabled: + if ( + not system_config.agent_conversation_memory_enabled + or self._assistant_memory_is_suppressed() + ): return messages, "" if role_config is None: return messages, "" @@ -1327,6 +1392,9 @@ async def _maybe_add_assistant_memory_context( task_id, prompt, workflow_mode_override=workflow_mode_override, + run_id_override=await self._assistant_run_id_for_workflow( + workflow_mode_override or self._assistant_workflow_mode_for_role(role_id) + ), ) target_hash = assistant_proof_search_coordinator.submit_target(snapshot) pack = assistant_proof_search_coordinator.get_latest_pack(target_hash) @@ -1370,7 +1438,10 @@ async def prewarm_assistant_memory_context( non-blocking Assistant lifecycle as normal completions, even if the prompt later overflows and no model call is made. """ - if not system_config.agent_conversation_memory_enabled: + if ( + not system_config.agent_conversation_memory_enabled + or self._assistant_memory_is_suppressed() + ): return "" async with self._state_lock: role_config = self._role_model_configs.get(role_id) @@ -1386,10 +1457,41 @@ async def prewarm_assistant_memory_context( task_id, prompt, workflow_mode_override=workflow_mode_override, + run_id_override=await self._assistant_run_id_for_workflow( + workflow_mode_override or self._assistant_workflow_mode_for_role(role_id) + ), ) target_hash = assistant_proof_search_coordinator.submit_target(snapshot) return target_hash + @staticmethod + async def _assistant_run_id_for_workflow(workflow_mode: str) -> str: + """Resolve durable workflow identity without deriving it from model prompts.""" + mode = str(workflow_mode or "").strip().lower() + if mode in {"aggregator", "compiler", "manual_proof_check"}: + try: + from backend.autonomous.memory.proof_database import manual_proof_database + return await manual_proof_database.get_or_create_active_run_id() + except Exception: + logger.debug("Could not resolve active manual Assistant run ID", exc_info=True) + return "" + if mode == "autonomous": + try: + from backend.autonomous.memory.session_manager import session_manager + return str(session_manager.session_id or "").strip() if session_manager.is_session_active else "" + except Exception: + logger.debug("Could not resolve autonomous Assistant run ID", exc_info=True) + return "" + if mode == "leanoj": + try: + from backend.leanoj.core.leanoj_coordinator import leanoj_coordinator + state = getattr(leanoj_coordinator, "_state", None) + return str(getattr(state, "session_id", "") or "").strip() + except Exception: + logger.debug("Could not resolve LeanOJ Assistant run ID", exc_info=True) + return "" + return "" + @staticmethod def _append_assistant_memory_block(prompt: str, assistant_context: str) -> str: return ( @@ -1397,7 +1499,9 @@ def _append_assistant_memory_block(prompt: str, assistant_context: str) -> str: "OPTIONAL ASSISTANT MEMORY CONTEXT:\n" f"{assistant_context}\n\n" "Use the Assistant memory only when it is relevant. It is supporting context, " - "not validator feedback, not a requirement to cite, and not a replacement for the user prompt." + "not validator feedback, not a requirement to cite, and not a replacement for the user prompt. " + "These proof supports cannot redirect or mathematically reinterpret the user objective, " + "and they do not require mathematics or formal proof." ) def _prompt_fits_role_budget( @@ -1682,6 +1786,50 @@ async def _generate_completion_once( Returns: API response dict """ + try: + return await self._generate_completion_once_routed( + task_id=task_id, + role_id=role_id, + model=model, + messages=messages, + temperature=temperature, + max_tokens=max_tokens, + response_format=response_format, + tools=tools, + tool_choice=tool_choice, + **kwargs, + ) + except (ProviderRouteError, ProviderContextLengthError) as error: + if error.route.role_id == role_id and error.route.task_id == task_id: + raise + route_kind = error.route.route_kind + if error.route.provider == "lm_studio": + configured = self._role_model_configs.get(role_id) + if configured and configured.provider != "lm_studio": + route_kind = "fallback" + configured = self._role_model_configs.get(role_id) + raise error.with_route_context( + role_id=role_id, + task_id=task_id, + route_kind=route_kind, + configured_provider=configured.provider if configured else "", + configured_model=configured.model_id if configured else model, + ) from error + + async def _generate_completion_once_routed( + self, + task_id: str, + role_id: str, + model: str, + messages: List[Dict[str, str]], + temperature: float = 0.0, + max_tokens: Optional[int] = None, + response_format: Optional[Dict[str, str]] = None, + tools: Optional[List[Dict[str, Any]]] = None, + tool_choice: Optional[Any] = None, + **kwargs + ) -> Dict[str, Any]: + """Execute one routed call; the public helper adds safe route context.""" forced_boost_mode = kwargs.pop("_moto_force_boost_mode", None) consume_boost_count = kwargs.pop("_moto_consume_boost_count", True) strict_boost = kwargs.pop("_moto_strict_boost", False) @@ -2004,6 +2152,31 @@ async def _generate_completion_once( raise RuntimeError(f"Strict boost call credits exhausted for task {task_id}: {e}") from e # Continue to primary model routing below + except ProviderContextLengthError as e: + duration_ms = (time.time() - start_time) * 1000 + await boost_logger.log_boost_call( + task_id=task_id, + role_id=role_id, + model=boost_model, + prompt_preview=prompt_preview, + response_content="", + duration_ms=duration_ms, + success=False, + error=e.safe_message, + boost_mode=boost_mode, + ) + typed_error = _typed_provider_context_error( + e, + provider="openrouter", + model=boost_model, + route_kind="boost", + ) + if strict_boost: + raise typed_error from e + logger.warning( + "Boost context limit rejected task %s; trying primary route", + task_id, + ) except Exception as e: # Log the failed boost call duration_ms = (time.time() - start_time) * 1000 @@ -2531,9 +2704,11 @@ async def _generate_completion_once( tokens_used=None, duration_ms=duration_ms, success=False, - error=str(e), + error=oauth_live_activity_error_message(e), phase=self._current_autonomous_phase, ) + if is_retryable_model_output_error(e): + raise if role_config.lm_studio_fallback_id: async with self._state_lock: self._role_fallback_state[role_id] = "lm_studio" @@ -2544,6 +2719,12 @@ async def _generate_completion_once( ) model = role_config.lm_studio_fallback_id else: + if _is_provider_context_length_error(e): + raise _typed_provider_context_error( + e, + provider="sakana_fugu", + model=sakana_model, + ) from e if is_transient_model_call_error(e): raise self._as_retryable_provider_error( provider="sakana_fugu", @@ -2552,10 +2733,6 @@ async def _generate_completion_once( model=sakana_model, error=e, ) from e - if _is_provider_context_length_error(e): - raise ValueError( - f"Sakana Fugu context_length_exceeded for role '{role_id}': {e}" - ) from e await self._broadcast_unrecoverable_sakana_fugu_error( role_id=role_id, model=sakana_model, @@ -2578,9 +2755,11 @@ async def _generate_completion_once( tokens_used=None, duration_ms=duration_ms, success=False, - error=str(e), + error=oauth_live_activity_error_message(e), phase=self._current_autonomous_phase, ) + if is_retryable_model_output_error(e): + raise if role_config.lm_studio_fallback_id: async with self._state_lock: self._role_fallback_state[role_id] = "lm_studio" @@ -2592,6 +2771,12 @@ async def _generate_completion_once( ) model = role_config.lm_studio_fallback_id else: + if _is_provider_context_length_error(e): + raise _typed_provider_context_error( + e, + provider="sakana_fugu", + model=sakana_model, + ) from e if is_transient_model_call_error(e): raise self._as_retryable_provider_error( provider="sakana_fugu", @@ -2600,10 +2785,6 @@ async def _generate_completion_once( model=sakana_model, error=e, ) from e - if _is_provider_context_length_error(e): - raise ValueError( - f"Sakana Fugu context_length_exceeded for role '{role_id}': {e}" - ) from e await self._broadcast_unrecoverable_sakana_fugu_error( role_id=role_id, model=sakana_model, @@ -2776,9 +2957,11 @@ async def _generate_completion_once( tokens_used=None, duration_ms=duration_ms, success=False, - error=str(e), + error=oauth_live_activity_error_message(e), phase=self._current_autonomous_phase, ) + if is_retryable_model_output_error(e): + raise if role_config.lm_studio_fallback_id: async with self._state_lock: self._role_fallback_state[role_id] = "lm_studio" @@ -2789,6 +2972,12 @@ async def _generate_completion_once( ) model = role_config.lm_studio_fallback_id else: + if _is_provider_context_length_error(e): + raise _typed_provider_context_error( + e, + provider="openai_codex_oauth", + model=codex_model, + ) from e if is_transient_model_call_error(e): raise self._as_retryable_provider_error( provider="openai_codex_oauth", @@ -2797,10 +2986,6 @@ async def _generate_completion_once( model=codex_model, error=e, ) from e - if _is_provider_context_length_error(e): - raise ValueError( - f"OpenAI Codex context_length_exceeded for role '{role_id}': {e}" - ) from e await self._broadcast_unrecoverable_codex_error( role_id=role_id, model=codex_model, @@ -2823,9 +3008,11 @@ async def _generate_completion_once( tokens_used=None, duration_ms=duration_ms, success=False, - error=str(e), + error=oauth_live_activity_error_message(e), phase=self._current_autonomous_phase, ) + if is_retryable_model_output_error(e): + raise if role_config.lm_studio_fallback_id: async with self._state_lock: self._role_fallback_state[role_id] = "lm_studio" @@ -2837,6 +3024,12 @@ async def _generate_completion_once( ) model = role_config.lm_studio_fallback_id else: + if _is_provider_context_length_error(e): + raise _typed_provider_context_error( + e, + provider="openai_codex_oauth", + model=codex_model, + ) from e if is_transient_model_call_error(e): raise self._as_retryable_provider_error( provider="openai_codex_oauth", @@ -2845,10 +3038,6 @@ async def _generate_completion_once( model=codex_model, error=e, ) from e - if _is_provider_context_length_error(e): - raise ValueError( - f"OpenAI Codex context_length_exceeded for role '{role_id}': {e}" - ) from e await self._broadcast_unrecoverable_codex_error( role_id=role_id, model=codex_model, @@ -2944,9 +3133,11 @@ async def _generate_completion_once( tokens_used=None, duration_ms=duration_ms, success=False, - error=str(e), + error=oauth_live_activity_error_message(e), phase=self._current_autonomous_phase, ) + if is_retryable_model_output_error(e): + raise if role_config.lm_studio_fallback_id: async with self._state_lock: self._role_fallback_state[role_id] = "lm_studio" @@ -2957,6 +3148,12 @@ async def _generate_completion_once( ) model = role_config.lm_studio_fallback_id else: + if _is_provider_context_length_error(e): + raise _typed_provider_context_error( + e, + provider="xai_grok_oauth", + model=xai_model, + ) from e if is_transient_model_call_error(e): raise self._as_retryable_provider_error( provider="xai_grok_oauth", @@ -2965,10 +3162,6 @@ async def _generate_completion_once( model=xai_model, error=e, ) from e - if _is_provider_context_length_error(e): - raise ValueError( - f"xAI Grok context_length_exceeded for role '{role_id}': {e}" - ) from e await self._broadcast_unrecoverable_xai_grok_error( role_id=role_id, model=xai_model, @@ -2991,9 +3184,11 @@ async def _generate_completion_once( tokens_used=None, duration_ms=duration_ms, success=False, - error=str(e), + error=oauth_live_activity_error_message(e), phase=self._current_autonomous_phase, ) + if is_retryable_model_output_error(e): + raise if role_config.lm_studio_fallback_id: async with self._state_lock: self._role_fallback_state[role_id] = "lm_studio" @@ -3005,6 +3200,12 @@ async def _generate_completion_once( ) model = role_config.lm_studio_fallback_id else: + if _is_provider_context_length_error(e): + raise _typed_provider_context_error( + e, + provider="xai_grok_oauth", + model=xai_model, + ) from e if is_transient_model_call_error(e): raise self._as_retryable_provider_error( provider="xai_grok_oauth", @@ -3013,10 +3214,6 @@ async def _generate_completion_once( model=xai_model, error=e, ) from e - if _is_provider_context_length_error(e): - raise ValueError( - f"xAI Grok context_length_exceeded for role '{role_id}': {e}" - ) from e await self._broadcast_unrecoverable_xai_grok_error( role_id=role_id, model=xai_model, @@ -3219,6 +3416,30 @@ async def _try_free_model_rotation( except CreditExhaustionError as inner_e: logger.warning(f"Rotated model {alt_model} credit exhaustion: {inner_e}") break + except OpenRouterPrivacyPolicyError: + raise + except ProviderContextLengthError as inner_e: + raise inner_e.with_route_context( + role_id=role_id, + task_id=task_id, + route_kind="free_rotation", + configured_provider=configured_provider, + configured_model=configured_model, + ) from inner_e + except OpenRouterNoEndpointsError as inner_e: + free_model_manager.mark_model_failed(alt_model) + logger.warning( + "Rotated model %s has no available endpoint, trying next: %s", + alt_model, + inner_e, + ) + except (OpenRouterInvalidResponseError, ValueError) as inner_e: + free_model_manager.mark_model_failed(alt_model) + logger.warning( + "Rotated model %s failed provider call, trying next: %s", + alt_model, + inner_e, + ) # Step 2: Auto-Selector Backup — try openrouter/free if free_model_manager.auto_selector_enabled: @@ -3266,6 +3487,14 @@ async def _try_free_model_rotation( if free_model_manager.is_account_exhausted(): free_model_manager.clear_account_exhaustion() return result + except ProviderContextLengthError as inner_e: + raise inner_e.with_route_context( + role_id=role_id, + task_id=task_id, + route_kind="auto_selector", + configured_provider=configured_provider, + configured_model=configured_model, + ) from inner_e except (RateLimitError, CreditExhaustionError) as inner_e: logger.warning(f"Auto-selector '{auto_model}' also failed: {inner_e}") diff --git a/backend/shared/api_log_persistence.py b/backend/shared/api_log_persistence.py new file mode 100644 index 0000000..5b7f0a2 --- /dev/null +++ b/backend/shared/api_log_persistence.py @@ -0,0 +1,48 @@ +"""Shared persistence helpers for metadata-only API logs.""" + +import os +import tempfile +from pathlib import Path +from typing import Iterable + + +def ensure_private_log_file(path: Path) -> None: + """Create a log file with owner-only POSIX permissions when supported.""" + path.parent.mkdir(parents=True, exist_ok=True) + if not path.exists(): + if os.name == "posix": + descriptor = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600) + os.close(descriptor) + else: + path.touch() + if os.name == "posix": + path.chmod(0o600) + + +def atomic_write_log_lines(path: Path, lines: Iterable[str]) -> None: + """Replace a JSONL log atomically without widening POSIX permissions.""" + ensure_private_log_file(path) + descriptor, temporary_name = tempfile.mkstemp( + prefix=f".{path.name}.", + suffix=".tmp", + dir=path.parent, + text=True, + ) + temporary_path = Path(temporary_name) + try: + if os.name == "posix": + os.chmod(temporary_path, 0o600) + with os.fdopen(descriptor, "w", encoding="utf-8", newline="") as handle: + handle.writelines(lines) + handle.flush() + os.fsync(handle.fileno()) + temporary_path.replace(path) + if os.name == "posix": + path.chmod(0o600) + except Exception: + try: + os.close(descriptor) + except OSError: + pass + temporary_path.unlink(missing_ok=True) + raise diff --git a/backend/shared/boost_logger.py b/backend/shared/boost_logger.py index 13df4ff..a69d7c0 100644 --- a/backend/shared/boost_logger.py +++ b/backend/shared/boost_logger.py @@ -8,12 +8,13 @@ import json import logging import os -from collections import deque +from collections import OrderedDict, deque from datetime import datetime from typing import Dict, Any, List, Optional from pathlib import Path from backend.shared.config import system_config +from backend.shared.api_log_persistence import atomic_write_log_lines, ensure_private_log_file from backend.shared.log_redaction import redact_log_text logger = logging.getLogger(__name__) @@ -52,17 +53,24 @@ def __init__(self): return self._initialized = True + self._prepared_root_identity = None + self._volatile_payloads: OrderedDict[str, Dict[str, str]] = OrderedDict() + self._prepare_active_root() + logger.info("BoostLogger initialized") + + def _prepare_active_root(self) -> None: + identity = system_config.runtime_root_identity() + if self._prepared_root_identity == identity: + return + self._volatile_payloads.clear() self._ensure_log_file() self._scrub_persisted_full_payloads() - logger.info("BoostLogger initialized") + self._prepared_root_identity = identity def _ensure_log_file(self) -> None: """Ensure the log file and directory exist.""" log_path = self._get_log_path() - log_path.parent.mkdir(parents=True, exist_ok=True) - - if not log_path.exists(): - log_path.write_text("") + ensure_private_log_file(log_path) def _get_log_path(self) -> Path: """Return the instance-scoped boost log path.""" @@ -117,8 +125,7 @@ def _scrub_persisted_full_payloads(self) -> None: scrubbed_lines.append(json.dumps(entry) + "\n") if changed: - with open(log_path, "w", encoding="utf-8") as f: - f.writelines(scrubbed_lines) + atomic_write_log_lines(log_path, scrubbed_lines) logger.info("Scrubbed legacy full payloads from boost API log") except Exception as e: logger.warning(f"Failed to scrub legacy boost API log payloads: {e}") @@ -153,10 +160,19 @@ async def log_boost_call( """ async with self._lock: try: + self._prepare_active_root() response_meta = _payload_metadata(response_content, 2000) - store_full_payloads = bool(system_config.api_log_store_full_payloads) + store_full_payloads = bool( + system_config.api_log_store_full_payloads + and not system_config.generic_mode + ) + timestamp = datetime.now().isoformat() + entry_id = hashlib.sha256( + f"{timestamp}\0{task_id}\0{role_id}".encode("utf-8", errors="replace") + ).hexdigest() log_entry = { - "timestamp": datetime.now().isoformat(), + "entry_id": entry_id, + "timestamp": timestamp, "task_id": task_id, "role_id": role_id, "model": model, @@ -165,17 +181,23 @@ async def log_boost_call( "response_preview": response_meta["preview"], "response_size": response_meta["size"], "response_sha256": response_meta["sha256"], - "response_redacted": not store_full_payloads, - "has_full_response": store_full_payloads and bool(response_content), + "response_redacted": True, + "has_full_response": False, "tokens_used": tokens_used, "duration_ms": duration_ms, "success": success, "error": redact_log_text(error, 1000) } if store_full_payloads: - log_entry["response_full"] = response_content + self._volatile_payloads[entry_id] = { + "response_full": response_content, + "workflow": self._entry_workflow(log_entry), + } + while len(self._volatile_payloads) > self.MAX_LOG_ENTRIES: + self._volatile_payloads.popitem(last=False) # Append to log file + ensure_private_log_file(self._get_log_path()) with open(self._get_log_path(), "a", encoding="utf-8") as f: f.write(json.dumps(log_entry) + "\n") @@ -196,8 +218,7 @@ async def _trim_log_if_needed(self) -> None: if len(lines) > self.MAX_LOG_ENTRIES: # Keep only the most recent entries lines = lines[-self.MAX_LOG_ENTRIES:] - with open(self._get_log_path(), "w", encoding="utf-8") as f: - f.writelines(lines) + atomic_write_log_lines(self._get_log_path(), lines) logger.debug(f"Trimmed boost log to {self.MAX_LOG_ENTRIES} entries") except Exception as e: @@ -215,6 +236,7 @@ async def get_logs(self, limit: int = 100, include_full: bool = True) -> List[Di """ async with self._lock: try: + self._prepare_active_root() log_path = self._get_log_path() if not os.path.exists(log_path): return [] @@ -228,15 +250,22 @@ async def get_logs(self, limit: int = 100, include_full: bool = True) -> List[Di if line: try: log_entry = json.loads(line) - if not include_full or not system_config.api_log_store_full_payloads: - prompt_full = str(log_entry.pop("prompt_full", "") or "") - response_full = str(log_entry.pop("response_full", "") or "") - log_entry["prompt_size"] = len(prompt_full) - log_entry["response_size"] = int(log_entry.get("response_size") or len(response_full)) - log_entry["has_full_prompt"] = False + prompt_full = str(log_entry.pop("prompt_full", "") or "") + response_full = str(log_entry.pop("response_full", "") or "") + log_entry["prompt_size"] = int(log_entry.get("prompt_size") or len(prompt_full)) + log_entry["response_size"] = int(log_entry.get("response_size") or len(response_full)) + cached = self._volatile_payloads.get(str(log_entry.get("entry_id") or "")) + if ( + include_full + and cached + and system_config.api_log_store_full_payloads + and not system_config.generic_mode + ): + log_entry["response_full"] = cached["response_full"] + log_entry["has_full_response"] = bool(cached["response_full"]) + else: log_entry["has_full_response"] = False - if response_full and not log_entry.get("response_sha256"): - log_entry["response_sha256"] = hashlib.sha256(response_full.encode("utf-8", errors="replace")).hexdigest() + log_entry["has_full_prompt"] = False logs.append(log_entry) except json.JSONDecodeError: continue @@ -280,6 +309,7 @@ async def clear_logs(self, workflow: Optional[str] = None) -> None: """Clear boost API logs, optionally scoped to one workflow.""" async with self._lock: try: + self._prepare_active_root() if workflow: log_path = self._get_log_path() if not os.path.exists(log_path): @@ -301,13 +331,17 @@ async def clear_logs(self, workflow: Optional[str] = None) -> None: if self._entry_workflow(entry) != workflow: retained_lines.append(line) - with open(log_path, "w", encoding="utf-8") as f: - f.writelines(retained_lines) + self._volatile_payloads = OrderedDict( + (entry_id, payload) + for entry_id, payload in self._volatile_payloads.items() + if payload.get("workflow") != workflow + ) + atomic_write_log_lines(log_path, retained_lines) logger.info("Boost logs cleared for workflow %s", workflow) return - with open(self._get_log_path(), "w", encoding="utf-8") as f: - f.write("") + self._volatile_payloads.clear() + atomic_write_log_lines(self._get_log_path(), []) logger.info("Boost logs cleared") except Exception as e: logger.error(f"Failed to clear boost logs: {e}") diff --git a/backend/shared/build_info.py b/backend/shared/build_info.py index d8019dd..9d4b0cc 100644 --- a/backend/shared/build_info.py +++ b/backend/shared/build_info.py @@ -25,7 +25,7 @@ "version": "0.0.0-dev", "build_commit": "dev", "update_channel": "main", - "api_contract_version": "build5-v54", + "api_contract_version": "build5-v73", } _ENV_OVERRIDES = { diff --git a/backend/shared/config.py b/backend/shared/config.py index ac2c3e2..3bccdee 100644 --- a/backend/shared/config.py +++ b/backend/shared/config.py @@ -304,7 +304,7 @@ class SystemConfig(BaseSettings): validation_alias=AliasChoices("MOTO_Z3_PATH", "Z3_PATH"), ) smt_timeout: int = Field( - default=30, + default=300, validation_alias=AliasChoices("MOTO_SMT_TIMEOUT", "SMT_TIMEOUT"), ) @@ -362,6 +362,92 @@ class SystemConfig(BaseSettings): # Session-based organization (preferred for new features) auto_sessions_base_dir: Optional[str] = None + runtime_root_generation: int = 0 + + _DERIVED_DATA_PATHS = { + "user_uploads_dir": ("user_uploads",), + "chroma_db_dir": ("chroma_db",), + "shared_training_file": ("rag_shared_training.txt",), + "compiler_outline_file": ("compiler_outline.txt",), + "compiler_paper_file": ("compiler_paper.txt",), + "compiler_rejections_file": ("compiler_last_10_rejections.txt",), + "compiler_acceptances_file": ("compiler_last_10_acceptances.txt",), + "compiler_declines_file": ("compiler_last_10_declines.txt",), + "auto_brainstorms_dir": ("auto_brainstorms",), + "auto_papers_dir": ("auto_papers",), + "auto_papers_archive_dir": ("auto_papers", "archive"), + "auto_research_metadata_file": ("auto_research_metadata.json",), + "auto_research_stats_file": ("auto_research_stats.json",), + "auto_workflow_state_file": ("auto_workflow_state.json",), + "auto_research_topic_rejections_file": ("auto_research_topic_rejections.txt",), + "auto_sessions_base_dir": ("auto_sessions",), + } + + @staticmethod + def _normalized_root(value: str | Path, label: str) -> Path: + raw = str(value or "").strip() + if not raw: + raise ValueError(f"{label} must be a non-empty path.") + return Path(raw).expanduser().resolve(strict=False) + + def runtime_root_identity(self) -> tuple[str, str, int]: + """Return the stable identity used by late-bound mutable stores.""" + return ( + str(self._normalized_root(self.data_dir, "data root")), + str(self._normalized_root(self.logs_dir or "", "logs root")), + self.runtime_root_generation, + ) + + def assert_path_in_data_root(self, path: str | Path, label: str = "mutable path") -> Path: + """Validate that a mutable path belongs to the currently bound data root.""" + root = self._normalized_root(self.data_dir, "data root") + candidate = Path(path).expanduser().resolve(strict=False) + try: + candidate.relative_to(root) + except ValueError as exc: + raise RuntimeError( + f"Stale {label} is outside the active data root: {candidate} (active root: {root})." + ) from exc + return candidate + + def bind_runtime_roots( + self, + data_root: str | Path, + logs_root: str | Path | None = None, + ) -> tuple[str, str, int]: + """Atomically bind and validate the complete mutable runtime path graph.""" + data_path = self._normalized_root(data_root, "data root") + logs_path = ( + self._normalized_root(logs_root, "logs root") + if logs_root is not None + else ( + Path("backend/logs").resolve(strict=False) + if data_path == Path("backend/data").resolve(strict=False) + else data_path / "_logs" + ) + ) + + derived = { + field_name: str(data_path.joinpath(*parts)) + for field_name, parts in self._DERIVED_DATA_PATHS.items() + } + lean_workspace = str(data_path / "lean4_workspace") + + # Validate the complete graph before mutating any field. + for field_name, candidate in {**derived, "lean4_workspace_dir": lean_workspace}.items(): + candidate_path = Path(candidate).resolve(strict=False) + try: + candidate_path.relative_to(data_path) + except ValueError as exc: + raise ValueError(f"Derived runtime path escaped data root: {field_name}") from exc + + self.data_dir = str(data_path) + self.logs_dir = str(logs_path) + for field_name, value in derived.items(): + setattr(self, field_name, value) + self.lean4_workspace_dir = lean_workspace + self.runtime_root_generation += 1 + return self.runtime_root_identity() @model_validator(mode="after") def _derive_instance_paths(self) -> "SystemConfig": @@ -434,3 +520,11 @@ def _join_data_path(*parts: str) -> str: rag_config = RAGConfig() system_config = SystemConfig() + +def bind_runtime_roots( + data_root: str | Path, + logs_root: str | Path | None = None, +) -> tuple[str, str, int]: + """Supported process-wide API for atomically rebinding mutable runtime roots.""" + return system_config.bind_runtime_roots(data_root, logs_root) + diff --git a/backend/shared/context_overflow.py b/backend/shared/context_overflow.py index 6c8aee9..8892a53 100644 --- a/backend/shared/context_overflow.py +++ b/backend/shared/context_overflow.py @@ -1,4 +1,9 @@ -"""Shared user-facing context overflow messages.""" +"""Shared user-facing context overflow messages and event metadata.""" + +from typing import Any, Optional + +from backend.shared.models import ModelConfig +from backend.shared.provider_errors import ProviderRouteIdentity CONTEXT_OVERFLOW_STOP_REASON = "context_overflow" CONTEXT_OVERFLOW_STOP_MESSAGE = ( @@ -10,3 +15,32 @@ CONTEXT_OVERFLOW_RESOLUTION = ( "Start a new session with a condensed prompt, or choose a model with a higher context limit." ) + + +def context_overflow_model_payload( + config: Optional[ModelConfig] = None, + *, + route: Optional[ProviderRouteIdentity] = None, +) -> dict[str, Any]: + """Return safe configured/effective model metadata for an overflow event.""" + payload: dict[str, Any] = {} + configured_model = route.configured_model if route else "" + configured_provider = route.configured_provider if route else "" + if not configured_model and config is not None: + configured_model = config.model_id + if not configured_provider and config is not None: + configured_provider = config.provider + if configured_model: + payload["configured_model"] = configured_model + if configured_provider: + payload["configured_provider"] = configured_provider + if route is not None: + if route.model: + payload["effective_model"] = route.model + if route.provider: + payload["effective_provider"] = route.provider + if route.host_provider: + payload["effective_host_provider"] = route.host_provider + if route.route_kind: + payload["route_kind"] = route.route_kind + return payload diff --git a/backend/shared/critique_prompts.py b/backend/shared/critique_prompts.py index a86cecf..efd5ec4 100644 --- a/backend/shared/critique_prompts.py +++ b/backend/shared/critique_prompts.py @@ -16,7 +16,7 @@ Evaluate this paper and provide: 1. NOVELTY (1-10): How original and innovative is this work? -2. CORRECTNESS (1-10): How mathematically/logically sound is the content? +2. CORRECTNESS (1-10): How factually, logically, mathematically, technically, or methodologically sound is the content, as appropriate to its claims and field? 3. IMPACT ON RELATED FIELD (1-10): How significant could this contribution be? For each category, provide the numeric rating (1-10) and detailed feedback explaining your assessment. diff --git a/backend/shared/free_model_manager.py b/backend/shared/free_model_manager.py index 7341695..0410e0a 100644 --- a/backend/shared/free_model_manager.py +++ b/backend/shared/free_model_manager.py @@ -18,6 +18,58 @@ FAILED_MODEL_EXPIRY = 300 # 5 minutes +def _coerce_modalities(value: Any) -> Set[str]: + """Normalize OpenRouter modality metadata to lowercase tokens.""" + if value is None: + return set() + if isinstance(value, str): + return { + token.strip().lower() + for token in value.replace(",", "+").split("+") + if token.strip() + } + if isinstance(value, list): + return {str(token).strip().lower() for token in value if str(token).strip()} + return set() + + +def supports_text_chat_model(model: Dict[str, Any]) -> bool: + """ + Return True when model metadata says it can accept text and return text. + + OpenRouter exposes modality data either as architecture.input/output_modalities + or a compact architecture.modality string such as "text->text" or + "text->image". If modality metadata is absent, keep the model out of + automatic rotation and free-only role lists because a non-chat model is worse + than no rotation. + """ + architecture = model.get("architecture") + if not isinstance(architecture, dict): + architecture = {} + + input_modalities = _coerce_modalities( + architecture.get("input_modalities") or model.get("input_modalities") + ) + output_modalities = _coerce_modalities( + architecture.get("output_modalities") or model.get("output_modalities") + ) + + modality = architecture.get("modality") or model.get("modality") + if isinstance(modality, str) and "->" in modality: + input_text, output_text = modality.split("->", 1) + input_modalities = input_modalities or _coerce_modalities(input_text) + output_modalities = output_modalities or _coerce_modalities(output_text) + + if not input_modalities or not output_modalities: + return False + if "text" not in input_modalities: + return False + if "text" not in output_modalities: + return False + + return True + + class FreeModelManager: """Singleton managing free model rotation and account exhaustion state.""" @@ -25,8 +77,8 @@ class FreeModelManager: AUTO_SELECTOR_CONTEXT = 0 def __init__(self): - self.looping_enabled: bool = True - self.auto_selector_enabled: bool = True + self.looping_enabled: bool = False + self.auto_selector_enabled: bool = False self._cached_free_models: List[Dict[str, Any]] = [] self._cache_timestamp: float = 0.0 @@ -54,10 +106,11 @@ def reset(self) -> None: def update_cached_models(self, models: List[Dict[str, Any]]) -> None: """ - Cache free models sorted by context_length descending. + Cache free text-chat models sorted by context_length descending. Called when /api/openrouter/models is fetched. """ free_models = [] + skipped_non_text = 0 for m in models: pricing = m.get("pricing", {}) try: @@ -68,14 +121,21 @@ def update_cached_models(self, models: List[Dict[str, Any]]) -> None: except (ValueError, TypeError): continue if is_free: - free_models.append(m) + if supports_text_chat_model(m): + free_models.append(m) + else: + skipped_non_text += 1 free_models.sort( key=lambda m: m.get("context_length", 0), reverse=True ) self._cached_free_models = free_models self._cache_timestamp = time.time() - logger.info(f"Cached {len(free_models)} free models for rotation") + logger.info( + "Cached %s free text models for rotation%s", + len(free_models), + f" (skipped {skipped_non_text} non-text free models)" if skipped_non_text else "", + ) def _cleanup_expired_failures(self) -> None: """Remove models from failed list if their expiry has passed.""" diff --git a/backend/shared/lm_studio_client.py b/backend/shared/lm_studio_client.py index 672c1d6..f16c062 100644 --- a/backend/shared/lm_studio_client.py +++ b/backend/shared/lm_studio_client.py @@ -20,6 +20,11 @@ from typing import List, Dict, Any, Optional, Tuple from backend.shared.config import rag_config, system_config from backend.shared.log_redaction import redact_log_text +from backend.shared.provider_errors import ( + ProviderContextLengthError, + ProviderRouteError, + ProviderRouteIdentity, +) import logging logger = logging.getLogger(__name__) @@ -541,10 +546,12 @@ async def _execute_completion_request( f"Input prompt too large! Prompt: ~{approx_tokens} tokens, " f"Model context limit: {context_limit} tokens." ) - raise ValueError( + raise ProviderContextLengthError( f"Prompt ({approx_tokens} tokens) exceeds model's context window ({context_limit} tokens). " - f"In LM Studio: Increase 'Context Length (n_ctx)' to at least {approx_tokens + 5000} tokens." - ) + f"In LM Studio: Increase 'Context Length (n_ctx)' to at least {approx_tokens + 5000} tokens.", + route=ProviderRouteIdentity(provider="lm_studio", model=model), + cause=e, + ) from e # Retry on transient 400 errors if attempt < max_retries: @@ -568,7 +575,12 @@ async def _execute_completion_request( ) raise - except (httpx.ConnectError, httpx.RemoteProtocolError, httpx.ReadError) as e: + except ( + httpx.ConnectError, + httpx.RemoteProtocolError, + httpx.ReadError, + httpx.TimeoutException, + ) as e: logger.error( "Connection error for model '%s': %s", redact_log_text(model, 160), @@ -577,7 +589,11 @@ async def _execute_completion_request( if attempt < max_retries: await asyncio.sleep(1.0 * (attempt + 1)) continue - raise + raise ProviderRouteError( + "LM Studio connection failed after internal retries.", + route=ProviderRouteIdentity(provider="lm_studio", model=model), + cause=e, + ) from e except Exception as e: logger.error("Failed to generate completion: %s", redact_log_text(e, 240)) diff --git a/backend/shared/model_error_utils.py b/backend/shared/model_error_utils.py index 538de2a..d75d668 100644 --- a/backend/shared/model_error_utils.py +++ b/backend/shared/model_error_utils.py @@ -1,12 +1,15 @@ """Helpers for distinguishing model availability failures from ordinary output errors.""" from __future__ import annotations +import httpx + from backend.shared.openrouter_client import ( CreditExhaustionError, FreeModelExhaustedError, OpenRouterInvalidResponseError, OpenRouterPrivacyPolicyError, ) +from backend.shared.provider_errors import ProviderContextLengthError, ProviderRouteError _NON_RETRYABLE_MODEL_ERROR_MARKERS = ( @@ -80,6 +83,18 @@ "upstream connect error", ) +_PROVIDER_CONTEXT_LENGTH_MARKERS = ( + "context_length_exceeded", + "context length exceeded", + "exceeds the context window", + "exceeded the context window", + "too large for the context window", + "maximum context size", + "input exceeds", + "request too large", + "prompt is too long", +) + def format_transient_provider_error(exc: Exception) -> str: """Return a checkpoint-preserving transient provider error message.""" @@ -99,8 +114,30 @@ def is_retryable_model_output_error(exc: Exception) -> bool: return any(all(marker in message for marker in markers) for markers in _RETRYABLE_OUTPUT_FAILURE_MARKERS) +def is_provider_context_length_error(exc: Exception) -> bool: + """Return true when a provider rejects a request before generation because input is too large.""" + if isinstance(exc, ProviderContextLengthError): + return True + if isinstance(exc, ProviderRouteError) and exc.cause is not None: + return is_provider_context_length_error(exc.cause) + message = str(exc or "").lower() + return any(marker in message for marker in _PROVIDER_CONTEXT_LENGTH_MARKERS) + + def is_transient_model_call_error(exc: Exception) -> bool: """Return true for provider/network failures that should not be treated as config errors.""" + if isinstance(exc, ProviderRouteError) and exc.cause is not None: + if isinstance( + exc.cause, + ( + httpx.ConnectError, + httpx.ReadError, + httpx.RemoteProtocolError, + httpx.TimeoutException, + ), + ): + return True + return is_transient_model_call_error(exc.cause) if isinstance(exc, OpenRouterInvalidResponseError): content_type = str(getattr(exc, "content_type", "") or "").lower() body_preview = str(getattr(exc, "body_preview", "") or "").lower() @@ -116,6 +153,10 @@ def is_transient_model_call_error(exc: Exception) -> bool: def is_non_retryable_model_error(exc: Exception) -> bool: """Return true when a model/API failure should halt workflow progress.""" + if isinstance(exc, ProviderContextLengthError): + return True + if isinstance(exc, ProviderRouteError) and exc.cause is not None: + return is_non_retryable_model_error(exc.cause) if isinstance( exc, ( diff --git a/backend/shared/models.py b/backend/shared/models.py index d86e74d..c917f2f 100644 --- a/backend/shared/models.py +++ b/backend/shared/models.py @@ -5,7 +5,7 @@ from datetime import datetime from typing import List, Dict, Optional, Any, Literal -from pydantic import AliasChoices, BaseModel, ConfigDict, Field +from pydantic import AliasChoices, BaseModel, ConfigDict, Field, model_validator DEFAULT_CONTEXT_WINDOW = 0 DEFAULT_MAX_OUTPUT_TOKENS = 0 @@ -138,8 +138,8 @@ class BoostConfig(BaseModel): class FreeModelSettings(BaseModel): """Settings for free model cooldown handling and rotation.""" - looping_enabled: bool = True - auto_selector_enabled: bool = True + looping_enabled: bool = False + auto_selector_enabled: bool = False class WorkflowTask(BaseModel): @@ -390,12 +390,24 @@ class BrainstormMetadata(BaseModel): topic_id: str topic_prompt: str status: Literal["in_progress", "complete"] = "in_progress" + # Current post-cleanup database size. submission_count: int = 0 + # Monotonic accepted-event count used for resume offsets and hard caps. + total_acceptances: int = 0 created_at: datetime = Field(default_factory=datetime.now) completed_at: Optional[datetime] = None last_activity: datetime = Field(default_factory=datetime.now) papers_generated: List[str] = Field(default_factory=list) + @model_validator(mode="before") + @classmethod + def _restore_legacy_total_acceptances(cls, data: Any) -> Any: + """Legacy metadata only stored the retained submission count.""" + if isinstance(data, dict) and "total_acceptances" not in data: + data = dict(data) + data["total_acceptances"] = max(0, int(data.get("submission_count") or 0)) + return data + class PaperMetadata(BaseModel): """Metadata for a completed or in-progress paper.""" @@ -421,10 +433,9 @@ class PaperMetadata(BaseModel): class TopicSelectionSubmission(BaseModel): """Submission from topic selection agent.""" - action: Literal["new_topic", "continue_existing", "combine_topics"] + action: Literal["new_topic", "continue_existing"] topic_id: Optional[str] = None # Required if action is continue_existing - topic_ids: List[str] = Field(default_factory=list) # Required if action is combine_topics - topic_prompt: str = "" # Required if action is new_topic or combine_topics + topic_prompt: str = "" # Required if action is new_topic reasoning: str @@ -705,6 +716,19 @@ class ProofAttemptFeedback(BaseModel): strategy: Literal["full_script", "tactic_script"] = "full_script" tactic_trace: List[str] = Field(default_factory=list) success: bool = False + configured_model: Optional[str] = None + configured_provider: Optional[str] = None + effective_model: Optional[str] = None + effective_provider: Optional[str] = None + overflow_origin: Optional[Literal["local_preflight", "provider"]] = None + prompt_tokens: Optional[int] = None + max_input_tokens: Optional[int] = None + + +ProofArtifactPurpose = Literal[ + "verified_occurrence", + "standalone_exact_duplicate_emphasis", +] class ProofRecord(BaseModel): @@ -717,11 +741,21 @@ class ProofRecord(BaseModel): source_type: Literal["brainstorm", "paper", "leanoj_subproof", "leanoj_final"] source_id: str source_title: str = "" + run_id: str = "" + user_prompt: str = "" solver: str = "Lean 4" lean_code: str novel: bool = False novelty_tier: str = "not_novel" novelty_reasoning: str = "" + independent_novelty_tier: str = "" + independent_novelty_reasoning: str = "" + exact_duplicate_proof_id: str = "" + exact_duplicate_run_id: str = "" + artifact_purpose: ProofArtifactPurpose = "verified_occurrence" + canonical_identity_version: str = "" + canonical_theorem_statement_hash: str = "" + canonical_lean_code_hash: str = "" verification_notes: str = "" attempt_count: int = 0 attempts: List[ProofAttemptFeedback] = Field(default_factory=list) @@ -730,6 +764,83 @@ class ProofRecord(BaseModel): created_at: datetime = Field(default_factory=datetime.now) +class ProofLibraryEntry(BaseModel): + """Stable metadata returned by archived proof-library endpoints.""" + library_id: str = "" + session_id: str = "" + proof_id: str + theorem_name: str = "" + theorem_statement: str = "" + formal_sketch: str = "" + source_type: str = "" + source_id: str = "" + source_title: str = "" + run_id: str = "" + user_prompt: str = "" + solver: str = "Lean 4" + novel: bool = False + novelty_tier: str = "not_novel" + novelty_reasoning: str = "" + independent_novelty_tier: str = "" + independent_novelty_reasoning: str = "" + exact_duplicate_proof_id: str = "" + exact_duplicate_run_id: str = "" + artifact_purpose: ProofArtifactPurpose = "verified_occurrence" + canonical_identity_version: str = "" + canonical_theorem_statement_hash: str = "" + canonical_lean_code_hash: str = "" + verification_notes: str = "" + attempt_count: int = 0 + created_at: str = "" + dependencies: List[ProofDependency] = Field(default_factory=list) + lean_code: str = "" + + +class ProofLibraryCounts(BaseModel): + total: int = 0 + listed: int = 0 + novel: int = 0 + duplicate_novel: int = 0 + not_novel: int = 0 + + +class ProofLibraryResponse(BaseModel): + proofs: List[ProofLibraryEntry] = Field(default_factory=list) + counts: ProofLibraryCounts = Field(default_factory=ProofLibraryCounts) + scope: Literal["autonomous", "manual"] + category: Literal["novel", "duplicate_novel", "not_novel", "all"] + + +class ProofCertificateResponse(BaseModel): + proof_id: str + theorem_statement: str + theorem_name: str = "" + lean_code: str = "" + solver: str = "Lean 4" + lean_version: str = "" + mathlib_commit: str = "" + verified_at: Optional[str] = None + source_type: str = "" + source_id: str = "" + source_title: str = "" + run_id: str = "" + user_prompt: str = "" + novel: bool = False + novelty_tier: str = "not_novel" + novelty_reasoning: str = "" + independent_novelty_tier: str = "" + independent_novelty_reasoning: str = "" + exact_duplicate_proof_id: str = "" + exact_duplicate_run_id: str = "" + artifact_purpose: ProofArtifactPurpose = "verified_occurrence" + canonical_identity_version: str = "" + canonical_theorem_statement_hash: str = "" + canonical_lean_code_hash: str = "" + attempt_count: int = 0 + solver_hints: List[str] = Field(default_factory=list) + dependencies: List[ProofDependency] = Field(default_factory=list) + + class ProofAttemptResult(BaseModel): """Outcome of one theorem proof-attempt loop.""" theorem_id: str @@ -752,6 +863,7 @@ class ProofStageResult(BaseModel): results: List[ProofAttemptResult] = Field(default_factory=list) had_error: bool = False error_message: str = "" + deferred_candidate_ids: List[str] = Field(default_factory=list) class ProofCheckRequest(BaseModel): diff --git a/backend/shared/openai_codex_client.py b/backend/shared/openai_codex_client.py index 95736b4..e3624b2 100644 --- a/backend/shared/openai_codex_client.py +++ b/backend/shared/openai_codex_client.py @@ -523,6 +523,15 @@ def _resolve_model_request(cls, model: str, reasoning_effort: Optional[str]) -> """Map user-facing Codex aliases onto the backend model id/request knobs.""" if model == cls.CODEX_SPARK_HIGH_MODEL_ID: return cls.CODEX_SPARK_MODEL_ID, "high" + # Some Codex catalogs expose named variants as selectable entries while + # the Responses backend accepts the base model plus a reasoning effort. + # Keep the selected alias in MOTO metadata, but normalize the request. + named_effort = { + "gpt-5.6-luna": "high", + "gpt-5.6-terra": "medium", + }.get(model) + if named_effort: + return "gpt-5.6-sol", named_effort return model, reasoning_effort @staticmethod diff --git a/backend/shared/openrouter_client.py b/backend/shared/openrouter_client.py index ad5edd7..8cb37c8 100644 --- a/backend/shared/openrouter_client.py +++ b/backend/shared/openrouter_client.py @@ -13,7 +13,13 @@ from typing import List, Dict, Any, Optional from backend.shared.config import system_config +from backend.shared.free_model_manager import supports_text_chat_model from backend.shared.log_redaction import redact_log_text +from backend.shared.provider_errors import ( + ProviderContextLengthError, + ProviderRouteError, + ProviderRouteIdentity, +) logger = logging.getLogger(__name__) @@ -204,6 +210,8 @@ async def list_models(self, free_only: bool = False, raise_on_error: bool = Fals # Filter for free models if requested if free_only: filtered_models = [] + free_model_count = 0 + skipped_non_text = 0 for model in models: pricing = model.get("pricing", {}) prompt_price = pricing.get("prompt", "0") @@ -213,12 +221,22 @@ async def list_models(self, free_only: bool = False, raise_on_error: bool = Fals try: is_free = float(prompt_price) == 0.0 and float(completion_price) == 0.0 if is_free: - filtered_models.append(model) + free_model_count += 1 + if supports_text_chat_model(model): + filtered_models.append(model) + else: + skipped_non_text += 1 except (ValueError, TypeError): # Skip models with invalid pricing data continue - logger.info(f"Filtered {len(filtered_models)} free models out of {len(models)} total") + logger.info( + "Filtered %s free text models out of %s free models (%s total%s)", + len(filtered_models), + free_model_count, + len(models), + f", skipped {skipped_non_text} non-text free models" if skipped_non_text else "", + ) return sorted(filtered_models, key=lambda m: m.get("name", m.get("id", ""))) # Return all models sorted by name @@ -708,6 +726,29 @@ async def _execute_completion_request( f"OpenRouter credits exhausted for model '{model}'. " f"Falling back to LM Studio." ) + + route = ProviderRouteIdentity( + provider="openrouter", + model=model, + host_provider=provider or "", + ) + if any( + marker in error_detail_lower + for marker in ( + "context_length_exceeded", + "context length exceeded", + "exceeds the context window", + "maximum context size", + "input exceeds", + "request too large", + "prompt is too long", + ) + ): + raise ProviderContextLengthError( + "OpenRouter rejected the request because its input exceeded the provider context limit.", + route=route, + cause=e, + ) from e logger.error( f"OpenRouter HTTP {e.response.status_code} error (attempt {attempt + 1}/{self.MAX_RETRIES}): " @@ -719,9 +760,18 @@ async def _execute_completion_request( await asyncio.sleep(self.RETRY_DELAY * (attempt + 1)) continue - raise ValueError(f"OpenRouter API error: {error_detail}") + raise ProviderRouteError( + f"OpenRouter API error: {error_detail}", + route=route, + cause=e, + ) from e - except (httpx.ConnectError, httpx.RemoteProtocolError, httpx.ReadError) as e: + except ( + httpx.ConnectError, + httpx.RemoteProtocolError, + httpx.ReadError, + httpx.TimeoutException, + ) as e: # Use repr(e) to get detailed exception info even if str(e) is empty error_type = type(e).__name__ error_detail = repr(e) if not str(e) else str(e) @@ -732,7 +782,15 @@ async def _execute_completion_request( if attempt < self.MAX_RETRIES - 1: await asyncio.sleep(self.RETRY_DELAY * (attempt + 1)) continue - raise ValueError(f"OpenRouter connection failed after {self.MAX_RETRIES} attempts: [{error_type}] {error_detail}") + raise ProviderRouteError( + f"OpenRouter connection failed after {self.MAX_RETRIES} attempts: [{error_type}] {error_detail}", + route=ProviderRouteIdentity( + provider="openrouter", + model=model, + host_provider=provider or "", + ), + cause=e, + ) from e except Exception as e: logger.error(f"OpenRouter unexpected error: {e}") diff --git a/backend/shared/path_safety.py b/backend/shared/path_safety.py index aa6b8b8..78821dc 100644 --- a/backend/shared/path_safety.py +++ b/backend/shared/path_safety.py @@ -39,3 +39,16 @@ def resolve_path_within_root(root: Path, *unsafe_parts: str) -> Path: raise ValueError("Resolved path escapes trusted root") return Path(candidate_real) + + +def resolve_filename_within_root( + root: Path, + filename: str, + label: str = "filename", +) -> Path: + """Resolve exactly one validated basename beneath a trusted root.""" + safe_filename = validate_single_path_component(filename, label) + basename = Path(safe_filename).name + if basename != safe_filename: + raise ValueError(f"Invalid {label}: {filename}") + return resolve_path_within_root(root, basename) diff --git a/backend/shared/proof_identity.py b/backend/shared/proof_identity.py new file mode 100644 index 0000000..04e9444 --- /dev/null +++ b/backend/shared/proof_identity.py @@ -0,0 +1,45 @@ +"""Versioned canonical identity for internally managed Lean proof artifacts.""" +from __future__ import annotations + +import hashlib +from dataclasses import dataclass + + +CANONICAL_PROOF_IDENTITY_VERSION = "moto-proof-identity-v1" + + +def canonicalize_theorem_statement(value: str) -> str: + """Collapse theorem-statement whitespace for exact identity matching.""" + return " ".join((value or "").split()) + + +def canonicalize_lean_code(value: str) -> str: + """Normalize Lean newlines and outer whitespace without changing its body.""" + return (value or "").replace("\r\n", "\n").replace("\r", "\n").strip() + + +def _sha256(value: str) -> str: + return hashlib.sha256(value.encode("utf-8")).hexdigest() + + +@dataclass(frozen=True) +class CanonicalProofIdentity: + version: str + theorem_statement_hash: str + lean_code_hash: str + + @property + def key(self) -> str: + return f"{self.version}:{self.theorem_statement_hash}:{self.lean_code_hash}" + + +def canonical_proof_identity( + theorem_statement: str, + lean_code: str, +) -> CanonicalProofIdentity: + """Build the canonical identity used by MOTO/manual/LeanOJ proof records.""" + return CanonicalProofIdentity( + version=CANONICAL_PROOF_IDENTITY_VERSION, + theorem_statement_hash=_sha256(canonicalize_theorem_statement(theorem_statement)), + lean_code_hash=_sha256(canonicalize_lean_code(lean_code)), + ) diff --git a/backend/shared/proof_search/assistant_cache.py b/backend/shared/proof_search/assistant_cache.py index 486d56d..f1a41eb 100644 --- a/backend/shared/proof_search/assistant_cache.py +++ b/backend/shared/proof_search/assistant_cache.py @@ -13,6 +13,7 @@ from backend.shared.config import system_config from backend.shared.proof_search.assistant_models import ( + ASSISTANT_PROOF_PACK_SCHEMA_VERSION, AssistantProofPack, AssistantTargetSnapshot, ) @@ -122,6 +123,8 @@ def load_cached_pack( pack = AssistantProofPack.model_validate_json(row["pack_json"]) except Exception: return None + if pack.schema_version != ASSISTANT_PROOF_PACK_SCHEMA_VERSION: + return None freshness = "cached" if pack.target_hash == target_hash else "stale-but-best-known" return pack.model_copy(update={"target_hash": target_hash, "freshness": freshness}) diff --git a/backend/shared/proof_search/assistant_coordinator.py b/backend/shared/proof_search/assistant_coordinator.py index 8a3f4a8..e02f982 100644 --- a/backend/shared/proof_search/assistant_coordinator.py +++ b/backend/shared/proof_search/assistant_coordinator.py @@ -5,14 +5,21 @@ import hashlib import json import logging +from collections import Counter from datetime import datetime, timezone from pathlib import Path from typing import Any, Awaitable, Callable from backend.shared.config import system_config from backend.shared.json_parser import parse_json +from backend.shared.proof_identity import canonical_proof_identity from backend.shared.proof_search.assistant_cache import AssistantCooldownState, AssistantRankCache, goal_hash_for_snapshot -from backend.shared.proof_search.assistant_models import AssistantProofPack, AssistantProofSupport, AssistantTargetSnapshot +from backend.shared.proof_search.assistant_models import ( + ASSISTANT_PROOF_PACK_SCHEMA_VERSION, + AssistantProofPack, + AssistantProofSupport, + AssistantTargetSnapshot, +) from backend.shared.proof_search.assistant_ranker import ranked_candidates_to_cache_rows, score_assistant_proof_candidates, select_assistant_proof_supports from backend.shared.proof_search.models import ProofSearchRequest, UnifiedProofSearchRecord, default_proof_search_corpora from backend.shared.proof_search.search_service import ProofSearchService, proof_search_service @@ -23,6 +30,7 @@ _ASSISTANT_CANDIDATE_POOL_TARGET = 64 _ASSISTANT_SHORTLIST_TARGET = 21 _ASSISTANT_FINAL_PACK_LIMIT = 7 +_ASSISTANT_EXACT_LINEAGE_LIMIT = 100_000 _ASSISTANT_SELECTION_MAX_OUTPUT_TOKENS = 4096 _ASSISTANT_SELECTION_CODE_PREVIEW_CHARS = 1200 _CURRENT_RUN_CORPUS_SCOPES = {"active", "current"} @@ -35,6 +43,9 @@ "Assistant memory skipped because no external proof-memory records were available; " "the Assistant only performs proof-memory retrieval for now." ) +_ASSISTANT_EXCLUDE_EXACT_DUPLICATE_EMPHASIS = ( + "assistant_exclude_standalone_exact_duplicate_emphasis" +) AssistantSelector = Callable[ [AssistantTargetSnapshot, list[AssistantProofSupport], str, str, str], @@ -77,6 +88,14 @@ def get_latest_pack(self, target_hash: str | None = None) -> AssistantProofPack return None return _drop_current_run_supports_from_pack(next(reversed(self._packs.values()))) + def get_latest_reusable_pack(self) -> AssistantProofPack | None: + """Return the latest in-memory pack without scheduling an Assistant refresh.""" + if self._latest_pack_target_hash: + latest = self.get_latest_pack(self._latest_pack_target_hash) + if latest is not None: + return latest + return self.get_latest_pack() + def get_status(self) -> dict[str, Any]: latest_pack = self.get_latest_pack() enabled_corpora = default_proof_search_corpora() if self.enabled else [] @@ -99,6 +118,64 @@ def get_status(self) -> dict[str, Any]: "disabled_reason": disabled_reason, } + def get_latest_pack_payload(self) -> dict[str, Any]: + """Return the latest Assistant pack in the same metadata-only shape used by the UI event.""" + if not self.enabled: + return { + "enabled": False, + "has_pack": False, + "results": [], + "disabled_reason": "Session History Memory is disabled.", + } + latest_pack = self.get_latest_pack() or _read_latest_assistant_pack() + if latest_pack is None: + return { + "enabled": self.enabled, + "has_pack": False, + "results": [], + "disabled_reason": "" if self.enabled else "Session History Memory is disabled.", + } + return { + "enabled": self.enabled, + "has_pack": True, + **_pack_event_payload(latest_pack, max_result_count=_ASSISTANT_FINAL_PACK_LIMIT), + } + + def get_support_lineage( + self, + *, + target_hash: str, + support_search_id: str, + offset: int, + limit: int, + ) -> dict[str, Any] | None: + """Return one bounded metadata-only occurrence page for a target support.""" + pack = self.get_latest_pack(target_hash) + if pack is None: + persisted = _read_latest_assistant_pack() + pack = persisted if persisted and persisted.target_hash == target_hash else None + if pack is None: + return None + support = next( + (item for item in pack.results if item.search_id == support_search_id), + None, + ) + if support is None: + return None + occurrences = list(support.occurrence_provenance) + total = max(support.occurrence_total, len(occurrences)) + page = occurrences[offset:offset + limit] + next_offset = offset + len(page) + return { + "target_hash": target_hash, + "support_search_id": support_search_id, + "occurrence_total": total, + "offset": offset, + "limit": limit, + "next_offset": next_offset if next_offset < total else None, + "occurrences": page, + } + def submit_target(self, snapshot: AssistantTargetSnapshot) -> str: target_hash = snapshot.stable_hash() snapshot = snapshot.model_copy(update={"target_hash": target_hash}) @@ -111,7 +188,7 @@ def submit_target(self, snapshot: AssistantTargetSnapshot) -> str: ) return target_hash cached_pack = self._load_cached_pack(snapshot) - if cached_pack is not None: + if cached_pack is not None and cached_pack.results: self._packs[target_hash] = cached_pack logger.info( "Assistant memory loaded cached pack for %s/%s (target=%s, results=%s, freshness=%s)", @@ -145,25 +222,37 @@ def submit_target(self, snapshot: AssistantTargetSnapshot) -> str: ) return target_hash if self._latest_pack_target_hash and not self._latest_pack_has_enough_receiver_reads(): - if cached_pack is None: - latest_pack = self.get_latest_pack(self._latest_pack_target_hash) or self.get_latest_pack() - if latest_pack is not None: - self._packs[target_hash] = latest_pack.model_copy( + reusable_pack = cached_pack if cached_pack and cached_pack.results else None + if reusable_pack is None: + latest_pack = self._packs.get(self._latest_pack_target_hash) + reusable_pack = _filter_pack_for_snapshot(latest_pack, snapshot) + if reusable_pack is None or not reusable_pack.results: + fallback_latest = next(reversed(self._packs.values()), None) + reusable_pack = _filter_pack_for_snapshot(fallback_latest, snapshot) + if reusable_pack is not None and reusable_pack.results: + if cached_pack is None: + self._packs[target_hash] = reusable_pack.model_copy( update={ "target_hash": target_hash, "freshness": "stale-but-best-known", "selection_mode": "stale-but-best-known", } ) + self._latest_pack_target_hash = target_hash + logger.info( + "Assistant memory refresh deferred for %s/%s (latest_target=%s receiver_reads=%s/%s)", + snapshot.workflow_mode, + snapshot.target_kind, + self._latest_pack_target_hash[:12], + self._latest_pack_consumption_count, + _ASSISTANT_PACK_REFRESH_RECEIVER_READS, + ) + return target_hash logger.info( - "Assistant memory refresh deferred for %s/%s (latest_target=%s receiver_reads=%s/%s)", + "Assistant memory pack reuse skipped for %s/%s because requesting-run filtering removed all supports", snapshot.workflow_mode, snapshot.target_kind, - self._latest_pack_target_hash[:12], - self._latest_pack_consumption_count, - _ASSISTANT_PACK_REFRESH_RECEIVER_READS, ) - return target_hash existing = self._tasks.get(target_hash) if existing and not existing.done(): logger.info( @@ -310,44 +399,31 @@ async def _refresh_pack(self, snapshot: AssistantTargetSnapshot) -> None: if await self._maybe_skip_for_cooldown(snapshot): return warnings: list[str] = [] - corpora = default_proof_search_corpora() - if not corpora: - corpora = ["moto", "manual", "leanoj"] - - records: list[UnifiedProofSearchRecord] = [] - seen_ids: set[str] = set() + enabled_corpora = default_proof_search_corpora() excluded_session_ids = [session_id for session_id in [_active_autonomous_session_id()] if session_id] - for query in _build_query_variants(snapshot): - if len(records) >= _ASSISTANT_CANDIDATE_POOL_TARGET: - break - try: - query_records = await self._service.search_candidate_pool( - ProofSearchRequest( - query=query, - goal_statement=snapshot.target_statement or snapshot.lean_template, - imports=snapshot.imports or ["Mathlib"], - dependency_names=snapshot.dependency_names, - corpora=corpora, - verified_only=True, - include_partial=False, - include_failed=False, - limit=_ASSISTANT_CANDIDATE_POOL_TARGET, - hydrate_lean_code=True, - ), - pool_limit=_ASSISTANT_CANDIDATE_POOL_TARGET - len(records), - exclude_corpus_scopes=sorted(_CURRENT_RUN_CORPUS_SCOPES), - exclude_session_ids=excluded_session_ids, - ) - query_records = _filter_current_run_records(query_records) - except Exception as exc: - logger.debug("Assistant proof search query failed: %s", exc) - warnings.append(f"Search query failed: {exc}") - continue - for record in query_records: - if record.search_id in seen_ids: - continue - seen_ids.add(record.search_id) - records.append(record) + excluded_run_ids = _current_run_ids(snapshot) + lanes = _assistant_retrieval_lane_specs(snapshot, enabled_corpora) + lane_results = await asyncio.gather(*[ + self._retrieve_lane( + snapshot=snapshot, + lane_name=lane_name, + corpora=corpora, + queries=queries, + excluded_session_ids=excluded_session_ids, + excluded_run_ids=excluded_run_ids, + warnings=warnings, + ) + for lane_name, corpora, queries in lanes + ]) + lane_records = { + lane_name: _filter_assistant_lane_records(records) + for (lane_name, _corpora, _queries), records in zip(lanes, lane_results) + } + records, retrieval_observability = _fuse_assistant_retrieval_lanes( + lane_records, + snapshot, + limit=_ASSISTANT_CANDIDATE_POOL_TARGET, + ) if not records: if warnings: @@ -397,20 +473,92 @@ async def _refresh_pack(self, snapshot: AssistantTargetSnapshot) -> None: candidate_stats = await asyncio.to_thread(self._cache.load_candidate_stats, snapshot.target_hash) shortlist = select_assistant_proof_supports(ranked_candidates, limit=_ASSISTANT_SHORTLIST_TARGET, candidate_stats=candidate_stats) if not shortlist: - await self._publish_pack(snapshot, [], warnings=warnings, selection_mode="no_candidates", candidate_count=len(records), shortlist_count=0, selection_reasoning="No verified candidate supports were found.") + retrieval_observability["shortlist_21"] = _stage_counts([]) + await self._publish_pack(snapshot, [], warnings=warnings, selection_mode="no_candidates", candidate_count=len(records), shortlist_count=0, selection_reasoning="No verified candidate supports were found.", retrieval_observability=retrieval_observability) return - await self._select_and_publish_assistant_pack(snapshot=snapshot, shortlist=shortlist, warnings=warnings, candidate_count=len(records)) + retrieval_observability["shortlist_21"] = _stage_counts_from_supports(shortlist) + await self._select_and_publish_assistant_pack(snapshot=snapshot, shortlist=shortlist, warnings=warnings, candidate_count=len(records), retrieval_observability=retrieval_observability) + + async def _retrieve_lane(self, *, snapshot: AssistantTargetSnapshot, lane_name: str, corpora: list[str], queries: list[str], excluded_session_ids: list[str], excluded_run_ids: list[str], warnings: list[str]) -> list[UnifiedProofSearchRecord]: + records: list[UnifiedProofSearchRecord] = [] + seen_ids: set[str] = set() + if lane_name == "exact_cross_prompt": + target_identity = canonical_proof_identity( + snapshot.target_statement, + snapshot.lean_template, + ) + statement_hashes = ( + [target_identity.theorem_statement_hash] + if snapshot.target_statement.strip() + else [] + ) + code_hashes = ( + [target_identity.lean_code_hash] + if snapshot.lean_template.strip() + else [] + ) + try: + exact_search = getattr(self._service, "exact_identity_neighborhood", None) + if exact_search is not None: + found = await exact_search( + theorem_statement_hashes=statement_hashes, + lean_code_hashes=code_hashes, + corpora=corpora, + exclude_run_ids=excluded_run_ids, + exclude_session_ids=excluded_session_ids, + limit=_ASSISTANT_EXACT_LINEAGE_LIMIT, + ) + return _filter_current_run_records(found, excluded_run_ids=excluded_run_ids) + except Exception: + logger.debug("Assistant exact neighborhood lane failed", exc_info=True) + warnings.append("exact_cross_prompt lane query failed.") + return [] + for query in queries: + if len(records) >= _ASSISTANT_CANDIDATE_POOL_TARGET: + break + try: + request = ProofSearchRequest(query=query, goal_statement=snapshot.target_statement or snapshot.lean_template, imports=snapshot.imports, dependency_names=snapshot.dependency_names, corpora=corpora, verified_only=True, include_partial=False, include_failed=False, limit=_ASSISTANT_CANDIDATE_POOL_TARGET, hydrate_lean_code=False) + try: + found = await self._service.search_candidate_pool( + request, + pool_limit=_ASSISTANT_CANDIDATE_POOL_TARGET - len(records), + exclude_corpus_scopes=sorted(_CURRENT_RUN_CORPUS_SCOPES), + exclude_session_ids=excluded_session_ids, + exclude_run_ids=excluded_run_ids, + ) + except TypeError as exc: + if "exclude_run_ids" not in str(exc): + raise + found = await self._service.search_candidate_pool( + request, + pool_limit=_ASSISTANT_CANDIDATE_POOL_TARGET - len(records), + exclude_corpus_scopes=sorted(_CURRENT_RUN_CORPUS_SCOPES), + exclude_session_ids=excluded_session_ids, + ) + found = _filter_current_run_records(found, excluded_run_ids=excluded_run_ids) + except Exception as exc: + logger.debug("Assistant %s lane query failed: %s", lane_name, exc) + warnings.append(f"{lane_name} lane query failed.") + continue + for record in found: + if record.search_id not in seen_ids: + seen_ids.add(record.search_id) + records.append(record) + return records - async def _select_and_publish_assistant_pack(self, *, snapshot: AssistantTargetSnapshot, shortlist: list[AssistantProofSupport], warnings: list[str], candidate_count: int) -> None: + async def _select_and_publish_assistant_pack(self, *, snapshot: AssistantTargetSnapshot, shortlist: list[AssistantProofSupport], warnings: list[str], candidate_count: int, retrieval_observability: dict[str, Any]) -> None: assistant_role_id = _assistant_role_id_for_snapshot(snapshot) assistant_model_id = "injected-assistant" if self._assistant_selector is not None else _assistant_model_id(assistant_role_id) if not assistant_model_id: warnings.append(f"Assistant role '{assistant_role_id}' is not configured.") - await self._publish_pack(snapshot, [], warnings=warnings, selection_mode="unavailable", assistant_role_id=assistant_role_id, assistant_model_id="", candidate_count=candidate_count, shortlist_count=len(shortlist), selection_reasoning="Configured Assistant role was unavailable.") + await self._publish_pack(snapshot, [], warnings=warnings, selection_mode="unavailable", assistant_role_id=assistant_role_id, assistant_model_id="", candidate_count=candidate_count, shortlist_count=len(shortlist), selection_reasoning="Configured Assistant role was unavailable.", retrieval_observability=retrieval_observability) return if _assistant_oauth_provider_is_cooling_down(assistant_role_id): - latest_pack = self.get_latest_pack() + latest_pack = _filter_pack_for_snapshot( + next(reversed(self._packs.values()), None), + snapshot, + ) if latest_pack and latest_pack.results: supports = latest_pack.results[:_ASSISTANT_FINAL_PACK_LIMIT] await self._publish_pack( @@ -426,6 +574,7 @@ async def _select_and_publish_assistant_pack(self, *, snapshot: AssistantTargetS candidate_count=candidate_count, shortlist_count=len(shortlist), selection_reasoning="Reused latest cached Assistant pack while OAuth provider cooldown is active.", + retrieval_observability=retrieval_observability, ) return if shortlist: @@ -443,6 +592,7 @@ async def _select_and_publish_assistant_pack(self, *, snapshot: AssistantTargetS candidate_count=candidate_count, shortlist_count=len(shortlist), selection_reasoning="Used deterministic proof-support shortlist while OAuth provider cooldown is active.", + retrieval_observability=retrieval_observability, ) return @@ -484,11 +634,13 @@ async def _select_and_publish_assistant_pack(self, *, snapshot: AssistantTargetS shortlist_count=len(shortlist), selection_reasoning="Assistant LLM selection failed; no proof supports were published.", broadcast_update=False, + retrieval_observability=retrieval_observability, ) return selected_supports = _supports_for_selected_ids(shortlist, selected_search_ids) - await self._publish_pack(snapshot, selected_supports, warnings=warnings, selection_mode="assistant_llm", assistant_role_id=assistant_role_id, assistant_model_id=assistant_model_id, candidate_count=candidate_count, shortlist_count=len(shortlist), selection_reasoning=selection_reasoning) + retrieval_observability["final_selected"] = _stage_counts_from_supports(selected_supports) + await self._publish_pack(snapshot, selected_supports, warnings=warnings, selection_mode="assistant_llm", assistant_role_id=assistant_role_id, assistant_model_id=assistant_model_id, candidate_count=candidate_count, shortlist_count=len(shortlist), selection_reasoning=selection_reasoning, retrieval_observability=retrieval_observability) def _next_assistant_task_id(self, workflow_mode: str) -> str: self._task_sequence += 1 @@ -504,41 +656,21 @@ async def _select_with_assistant(self, snapshot: AssistantTargetSnapshot, shortl raise RuntimeError(f"Assistant role '{assistant_role_id}' is not configured") max_tokens = _assistant_selection_max_tokens(role_config.max_output_tokens) prompt = _build_assistant_selection_prompt(snapshot, shortlist) - try: - payload = await _generate_assistant_selection_payload( - prompt=prompt, - task_id=task_id, - assistant_role_id=assistant_role_id, - assistant_model_id=assistant_model_id, - max_tokens=max_tokens, - ) - selected_ids = _extract_valid_selected_search_ids(payload, shortlist) - except _AssistantSelectionOutputError as first_error: - repair_prompt = _build_assistant_selection_repair_prompt( - snapshot, - shortlist, - error=str(first_error), - ) - try: - payload = await _generate_assistant_selection_payload( - prompt=repair_prompt, - task_id=f"{task_id}_retry", - assistant_role_id=assistant_role_id, - assistant_model_id=assistant_model_id, - max_tokens=max_tokens, - ) - selected_ids = _extract_valid_selected_search_ids(payload, shortlist) - except _AssistantSelectionOutputError as retry_error: - raise _AssistantSelectionOutputError( - f"{first_error}; retry failed: {retry_error}" - ) from retry_error + payload = await _generate_assistant_selection_payload( + prompt=prompt, + task_id=task_id, + assistant_role_id=assistant_role_id, + assistant_model_id=assistant_model_id, + max_tokens=max_tokens, + ) + selected_ids = _extract_valid_selected_search_ids(payload, shortlist) clean_ids = [str(item).strip() for item in selected_ids if str(item).strip()] reasoning = str(payload.get("reasoning") or payload.get("selection_reasoning") or "").strip() or "Assistant selected proof supports for the current target." reasoning = _compact_for_assistant_selection(reasoning, 300) return clean_ids[:_ASSISTANT_FINAL_PACK_LIMIT], reasoning - async def _publish_pack(self, snapshot: AssistantTargetSnapshot, supports: list[AssistantProofSupport], *, warnings: list[str], selection_mode: str, assistant_role_id: str = "", assistant_model_id: str = "", candidate_count: int, shortlist_count: int, selection_reasoning: str = "", broadcast_update: bool = True) -> None: - pack = AssistantProofPack(workflow_mode=snapshot.workflow_mode, target_kind=snapshot.target_kind, target_hash=snapshot.target_hash, query_summary=_compact_query_summary(snapshot), results=supports[:_ASSISTANT_FINAL_PACK_LIMIT], warnings=warnings, selection_mode=selection_mode, assistant_role_id=assistant_role_id, assistant_model_id=assistant_model_id, candidate_count=candidate_count, shortlist_count=shortlist_count, selection_reasoning=selection_reasoning) + async def _publish_pack(self, snapshot: AssistantTargetSnapshot, supports: list[AssistantProofSupport], *, warnings: list[str], selection_mode: str, assistant_role_id: str = "", assistant_model_id: str = "", candidate_count: int, shortlist_count: int, selection_reasoning: str = "", broadcast_update: bool = True, retrieval_observability: dict[str, Any] | None = None) -> None: + pack = AssistantProofPack(workflow_mode=snapshot.workflow_mode, target_kind=snapshot.target_kind, target_hash=snapshot.target_hash, query_summary=_compact_query_summary(snapshot), results=supports[:_ASSISTANT_FINAL_PACK_LIMIT], warnings=warnings, selection_mode=selection_mode, assistant_role_id=assistant_role_id, assistant_model_id=assistant_model_id, candidate_count=candidate_count, shortlist_count=shortlist_count, selection_reasoning=selection_reasoning, retrieval_observability=retrieval_observability or {}) source_counts: dict[str, int] = {} for support in pack.results: source_counts[support.corpus] = source_counts.get(support.corpus, 0) + 1 @@ -564,7 +696,16 @@ async def _publish_pack(self, snapshot: AssistantTargetSnapshot, supports: list[ await asyncio.to_thread(self._cache.record_pack, snapshot=snapshot, pack=pack, selected_search_ids=[support.search_id for support in pack.results]) await self._persist_pack(pack) if broadcast_update: - await self._broadcast_event("assistant_proof_pack_updated", {"target_hash": pack.target_hash, "workflow_mode": pack.workflow_mode, "target_kind": pack.target_kind, "result_count": len(pack.results), "local_result_count": sum(count for corpus, count in source_counts.items() if corpus != "syntheticlib4"), "syntheticlib4_result_count": source_counts.get("syntheticlib4", 0), "source_counts": source_counts, "max_result_count": _ASSISTANT_FINAL_PACK_LIMIT, "workflow_phase": snapshot.workflow_phase, "source_type": snapshot.source_type, "source_id": snapshot.source_id, "warnings": pack.warnings[:3], "selection_mode": pack.selection_mode, "assistant_role_id": pack.assistant_role_id, "assistant_model_id": pack.assistant_model_id, "candidate_count": pack.candidate_count, "shortlist_count": pack.shortlist_count}) + await self._broadcast_event( + "assistant_proof_pack_updated", + _pack_event_payload( + pack, + max_result_count=_ASSISTANT_FINAL_PACK_LIMIT, + workflow_phase=snapshot.workflow_phase, + source_type=snapshot.source_type, + source_id=snapshot.source_id, + ), + ) await self._record_cooldown_outcome(snapshot, pack) def _on_task_done(self, target_hash: str, task: asyncio.Task) -> None: @@ -601,11 +742,14 @@ def _load_cached_pack(self, snapshot: AssistantTargetSnapshot) -> AssistantProof in_memory_pack = self._packs.get(previous_target_hash) if in_memory_pack is not None: freshness = "cached" if previous_target_hash == snapshot.target_hash else "stale-but-best-known" - return _drop_current_run_supports_from_pack(in_memory_pack.model_copy(update={"target_hash": snapshot.target_hash, "freshness": freshness, "selection_mode": freshness})) + return _filter_pack_for_snapshot( + in_memory_pack.model_copy(update={"target_hash": snapshot.target_hash, "freshness": freshness, "selection_mode": freshness}), + snapshot, + ) cached = self._cache.load_cached_pack(target_hash=snapshot.target_hash, goal_hash=goal_hash) if cached is not None: cached = cached.model_copy(update={"selection_mode": cached.freshness}) - return _drop_current_run_supports_from_pack(cached) + return _filter_pack_for_snapshot(cached, snapshot) except Exception: logger.debug("Assistant proof-search cache lookup failed", exc_info=True) return None @@ -615,6 +759,46 @@ def _cached_pack_is_reusable(pack: AssistantProofPack) -> bool: return pack.freshness == "cached" and bool(pack.results) +def _pack_event_payload( + pack: AssistantProofPack, + *, + max_result_count: int, + workflow_phase: str = "", + source_type: str = "", + source_id: str = "", +) -> dict[str, Any]: + source_counts: dict[str, int] = {} + for support in pack.results: + source_counts[support.corpus] = source_counts.get(support.corpus, 0) + 1 + return { + "target_hash": pack.target_hash, + "workflow_mode": pack.workflow_mode, + "target_kind": pack.target_kind, + "result_count": len(pack.results), + "local_result_count": sum( + count for corpus, count in source_counts.items() if corpus != "syntheticlib4" + ), + "syntheticlib4_result_count": source_counts.get("syntheticlib4", 0), + "source_counts": source_counts, + "max_result_count": max_result_count, + "workflow_phase": workflow_phase, + "source_type": source_type, + "source_id": source_id, + "warnings": pack.warnings[:3], + "selection_mode": pack.selection_mode, + "assistant_role_id": pack.assistant_role_id, + "assistant_model_id": pack.assistant_model_id, + "candidate_count": pack.candidate_count, + "shortlist_count": pack.shortlist_count, + "selection_reasoning": pack.selection_reasoning, + "query_summary": pack.query_summary, + "freshness": pack.freshness, + "retrieval_observability": pack.retrieval_observability, + "created_at": pack.created_at, + "results": [support.event_preview_dump() for support in pack.results], + } + + def _build_query_variants(snapshot: AssistantTargetSnapshot) -> list[str]: variants = [snapshot.search_text(), "\n\n".join(part for part in [snapshot.user_prompt, snapshot.current_prompt_or_topic, snapshot.writing_goal, snapshot.outline_summary] if part), "\n\n".join(part for part in [snapshot.user_prompt, snapshot.target_statement] if part), "\n\n".join(part for part in [snapshot.lean_template, snapshot.lean_error] if part), "\n\n".join(part for part in [snapshot.rejection_feedback, snapshot.proof_attempt_feedback] if part), "\n\n".join(part for part in [snapshot.accepted_memory_summary, snapshot.paper_or_proof_draft_summary, snapshot.recent_activity_summary] if part), " ".join([*snapshot.dependency_names, *snapshot.imports]), " ".join(snapshot.source_titles), snapshot.source_title] cleaned: list[str] = [] @@ -628,6 +812,153 @@ def _build_query_variants(snapshot: AssistantTargetSnapshot) -> list[str]: return cleaned or [snapshot.target_statement or snapshot.user_prompt or snapshot.lean_template] +def _assistant_retrieval_lane_specs( + snapshot: AssistantTargetSnapshot, + enabled_corpora: list[str], +) -> list[tuple[str, list[str], list[str]]]: + lanes: list[tuple[str, list[str], list[str]]] = [] + local = [corpus for corpus in ("moto", "manual", "leanoj") if corpus in enabled_corpora] + if local: + lanes.append(("local_history", local, _build_query_variants(snapshot))) + exact_queries = [ + value + for value in ( + snapshot.target_statement, + snapshot.lean_template, + " ".join(snapshot.dependency_names), + "\n".join((snapshot.user_prompt, snapshot.target_statement)), + ) + if value and value.strip() + ] + lanes.append(("exact_cross_prompt", local, exact_queries or _build_query_variants(snapshot)[:1])) + if "syntheticlib4" in enabled_corpora: + lanes.append(("syntheticlib4", ["syntheticlib4"], _build_query_variants(snapshot))) + return lanes + + +def _artifact_key(record: UnifiedProofSearchRecord) -> str: + if record.theorem_statement_hash and record.lean_code_hash: + return f"hash:{record.theorem_statement_hash}:{record.lean_code_hash}" + if record.external_fingerprint: + return f"fingerprint:{record.external_fingerprint}" + return f"search:{record.search_id}" + + +def _occurrence(record: UnifiedProofSearchRecord) -> dict[str, str]: + occurrence = { + "search_id": record.search_id, + "corpus": record.corpus, + "corpus_scope": record.corpus_scope or record.release_id, + "session_id": record.session_id, + "run_id": record.run_id, + "source_type": record.source_type, + "source_id": record.source_id, + "source_title": record.source_title, + } + if _is_standalone_exact_duplicate_emphasis_record(record): + occurrence[_ASSISTANT_EXCLUDE_EXACT_DUPLICATE_EMPHASIS] = "true" + return occurrence + + +def _fuse_assistant_retrieval_lanes( + lane_records: dict[str, list[UnifiedProofSearchRecord]], + snapshot: AssistantTargetSnapshot, + *, + limit: int, +) -> tuple[list[UnifiedProofSearchRecord], dict[str, Any]]: + scored_by_lane = { + lane: score_assistant_proof_candidates(records, snapshot) + for lane, records in lane_records.items() + } + # Reciprocal-rank fusion aggregates every lane contribution for an exact + # artifact. Content scores choose the best representative; they do not + # masquerade as an additional fusion vote. + contributions: dict[str, float] = {} + appearances: list[tuple[str, str, int, float, UnifiedProofSearchRecord]] = [] + for lane, ranked in scored_by_lane.items(): + best_lane_rank_by_key: dict[str, int] = {} + for rank, candidate in enumerate(ranked): + key = _artifact_key(candidate.record) + best_lane_rank_by_key.setdefault(key, rank) + appearances.append((key, lane, rank, candidate.score, candidate.record)) + for key, rank in best_lane_rank_by_key.items(): + contributions[key] = contributions.get(key, 0.0) + 1.0 / (rank + 1) + appearances.sort(key=lambda item: (-item[3], item[1], item[2], item[4].search_id)) + representatives: dict[str, UnifiedProofSearchRecord] = {} + lanes_by_key: dict[str, set[str]] = {} + occurrences_by_key: dict[str, list[dict[str, str]]] = {} + for key, lane, _, _, record in appearances: + representatives.setdefault(key, record) + lanes_by_key.setdefault(key, set()).add(lane) + occurrence = _occurrence(record) + if occurrence not in occurrences_by_key.setdefault(key, []): + occurrences_by_key[key].append(occurrence) + ordered_keys = sorted( + representatives, + key=lambda key: ( + -contributions[key], + -max(item[3] for item in appearances if item[0] == key), + representatives[key].search_id, + ), + ) + selected: list[UnifiedProofSearchRecord] = [] + for key in ordered_keys: + representative = representatives[key] + metadata = dict(representative.metadata) + metadata["assistant_retrieval_lanes"] = sorted(lanes_by_key[key]) + occurrences = sorted( + occurrences_by_key[key], + key=lambda item: ( + item.get("corpus", ""), + item.get("run_id", ""), + item.get("session_id", ""), + item.get("source_id", ""), + item.get("search_id", ""), + ), + ) + metadata["assistant_occurrences"] = occurrences + metadata["assistant_occurrence_total"] = len(occurrences) + metadata["assistant_occurrence_omitted"] = 0 + metadata["assistant_reciprocal_rank_score"] = contributions[key] + selected.append(representative.model_copy(update={"metadata": metadata})) + if len(selected) >= limit: + break + observability = { + "raw_by_lane": {lane: _stage_counts(records) for lane, records in lane_records.items()}, + "raw_total": _stage_counts([record for records in lane_records.values() for record in records]), + "fused_before_dedupe": _stage_counts([item[4] for item in appearances]), + "unique_records": _stage_counts(list({record.search_id: record for records in lane_records.values() for record in records}.values())), + "deduped_artifacts": _stage_counts(list(representatives.values())), + "deduped_distinct": _stage_counts(list(representatives.values())), + "fused_cap_64": _stage_counts(selected), + "matching_runs_examined": len({ + (record.corpus, record.session_id or record.corpus_scope, record.source_id) + for records in lane_records.values() for record in records + }), + "matching_occurrences_examined": sum(len(records) for records in lane_records.values()), + "matching_runs": len({ + record.run_id or record.session_id or record.source_id + for records in lane_records.values() for record in records + if record.run_id or record.session_id or record.source_id + }), + } + return selected, observability + + +def _stage_counts(records: list[UnifiedProofSearchRecord]) -> dict[str, Any]: + return { + "total": len(records), + "by_corpus": dict(sorted(Counter(record.corpus for record in records).items())), + } + + +def _stage_counts_from_supports(supports: list[AssistantProofSupport]) -> dict[str, Any]: + return { + "total": len(supports), + "by_corpus": dict(sorted(Counter(support.corpus for support in supports).items())), + } + + def _assistant_role_id_for_snapshot(snapshot: AssistantTargetSnapshot) -> str: workflow_mode = (snapshot.workflow_mode or "").lower() if workflow_mode == "manual_proof_check": @@ -730,16 +1061,29 @@ def _extract_valid_selected_search_ids(payload: dict[str, Any], shortlist: list[ return [] valid_ids = {support.search_id for support in shortlist} proof_id_to_search_ids: dict[str, list[str]] = {} + occurrence_id_to_search_ids: dict[str, list[str]] = {} for support in shortlist: proof_id = str(support.proof_id or "").strip() if proof_id: proof_id_to_search_ids.setdefault(proof_id, []).append(support.search_id) + for occurrence in support.occurrence_provenance: + occurrence_search_id = str(occurrence.get("search_id") or "").strip() + if occurrence_search_id: + occurrence_id_to_search_ids.setdefault( + occurrence_search_id, [] + ).append(support.search_id) normalized_ids: list[str] = [] invalid_ids: list[str] = [] for search_id in clean_ids: if search_id in valid_ids: normalized_ids.append(search_id) continue + occurrence_matches = list(dict.fromkeys( + occurrence_id_to_search_ids.get(search_id, []) + )) + if len(occurrence_matches) == 1: + normalized_ids.append(occurrence_matches[0]) + continue proof_id_matches = proof_id_to_search_ids.get(search_id, []) if len(proof_id_matches) == 1: normalized_ids.append(proof_id_matches[0]) @@ -799,7 +1143,14 @@ def _assistant_selection_prompt( return ( f"{prefix}\n" 'Required schema: {"selected_search_ids":["<exact SELECT_ID>"],"reasoning":"<=160 chars"}\n' - f"Rules: select at most {_ASSISTANT_FINAL_PACK_LIMIT}; copy only exact SELECT_ID values; do not return proof_id/display IDs; use [] if no listed proof support is genuinely useful for the target; no markdown.\n\n" + f"Rules: select at most {_ASSISTANT_FINAL_PACK_LIMIT}; copy only exact SELECT_ID values; do not return proof_id/display IDs; no markdown. " + "Select a Lean-verified theorem record only when its exact statement or proof structure has " + "a concrete transfer path to the unchanged user objective/current task. Shared keywords and " + "loose analogies are insufficient. Never reinterpret or narrow the target to make a proof " + "appear relevant, and do not require mathematics or formal proof. For a non-mathematical " + "target, use [] unless an explicit bound, invariant, impossibility result, optimization or " + "algorithmic structure, correctness argument, safety argument, or similarly concrete " + "mathematical result materially helps. Use [] whenever no listed support genuinely helps.\n\n" f"TARGET:\n{target}\n\n" f"CANDIDATES:\n{ids}\n" ) @@ -815,6 +1166,22 @@ def _format_assistant_candidate(support: AssistantProofSupport) -> str: ] if statement: parts.append(f" statement: {_compact_for_assistant_selection(statement, 220)}") + if support.proof_description: + parts.append( + f" description: {_compact_for_assistant_selection(support.proof_description, 220)}" + ) + if support.source_title: + parts.append( + f" source: {_compact_for_assistant_selection(support.source_title, 140)}" + ) + if support.dependency_names: + dependencies = ", ".join(support.dependency_names) + parts.append( + f" dependencies: {_compact_for_assistant_selection(dependencies, 180)}" + ) + if support.imports: + imports = ", ".join(support.imports) + parts.append(f" imports: {_compact_for_assistant_selection(imports, 140)}") return "\n".join(parts) @@ -848,19 +1215,142 @@ def _compact_query_summary(snapshot: AssistantTargetSnapshot) -> str: return summary[:600] + ("..." if len(summary) > 600 else "") -def _filter_current_run_records(records: list[UnifiedProofSearchRecord]) -> list[UnifiedProofSearchRecord]: - return [record for record in records if not _is_current_run_record(record)] +def _filter_current_run_records( + records: list[UnifiedProofSearchRecord], + *, + excluded_run_ids: list[str] | None = None, +) -> list[UnifiedProofSearchRecord]: + excluded = set(excluded_run_ids or []) + return [ + record for record in records + if not _is_current_run_record(record) and (not record.run_id or record.run_id not in excluded) + ] + + +def _is_standalone_exact_duplicate_emphasis_record( + record: UnifiedProofSearchRecord, +) -> bool: + """Exclude only records explicitly marked as duplicate-emphasis artifacts.""" + return record.metadata.get(_ASSISTANT_EXCLUDE_EXACT_DUPLICATE_EMPHASIS) is True -def _drop_current_run_supports_from_pack(pack: AssistantProofPack | None) -> AssistantProofPack | None: +def _filter_assistant_lane_records( + records: list[UnifiedProofSearchRecord], +) -> list[UnifiedProofSearchRecord]: + return [ + record + for record in records + if not _is_standalone_exact_duplicate_emphasis_record(record) + ] + + +def _sanitize_duplicate_emphasis_support( + support: AssistantProofSupport, + *, + excluded_run_ids: set[str] | None = None, + excluded_session_ids: set[str] | None = None, +) -> AssistantProofSupport | None: + """Remove ineligible occurrences while retaining any eligible fused occurrence.""" + excluded_runs = excluded_run_ids or set() + excluded_sessions = excluded_session_ids or set() + occurrences = list(support.occurrence_provenance) + eligible = [ + occurrence + for occurrence in occurrences + if occurrence.get(_ASSISTANT_EXCLUDE_EXACT_DUPLICATE_EMPHASIS) != "true" + and ( + occurrence.get("corpus_scope", "").strip().lower() + not in _CURRENT_RUN_CORPUS_SCOPES + ) + and ( + not occurrence.get("run_id") + or occurrence.get("run_id") not in excluded_runs + ) + and ( + not occurrence.get("session_id") + or occurrence.get("session_id") not in excluded_sessions + ) + ] + if occurrences and not eligible: + return None + if len(eligible) == len(occurrences): + return support + + representative = eligible[0] + prior_total = max(support.occurrence_total, len(occurrences)) + removed = len(occurrences) - len(eligible) + updates: dict[str, Any] = { + "occurrence_provenance": eligible, + "occurrence_total": max(len(eligible), prior_total - removed), + "occurrence_omitted": max(0, support.occurrence_omitted), + } + for field in ( + "search_id", + "corpus", + "corpus_scope", + "session_id", + "run_id", + "source_type", + "source_id", + "source_title", + ): + value = representative.get(field) + if value: + updates[field] = value + return support.model_copy(update=updates) + + +def _drop_current_run_supports_from_pack( + pack: AssistantProofPack | None, + *, + excluded_run_ids: list[str] | None = None, + excluded_session_ids: list[str] | None = None, +) -> AssistantProofPack | None: if pack is None or not pack.results: return pack - filtered_results = [support for support in pack.results if not _is_current_run_support(support)] - if len(filtered_results) == len(pack.results): + if pack.schema_version != ASSISTANT_PROOF_PACK_SCHEMA_VERSION: + return None + excluded_runs = set(excluded_run_ids or []) + excluded_sessions = set(excluded_session_ids or []) + filtered_results: list[AssistantProofSupport] = [] + changed = False + for support in pack.results: + sanitized = _sanitize_duplicate_emphasis_support( + support, + excluded_run_ids=excluded_runs, + excluded_session_ids=excluded_sessions, + ) + if sanitized is None: + changed = True + continue + if not support.occurrence_provenance and ( + _is_current_run_support(support) + or (support.run_id and support.run_id in excluded_runs) + or (support.session_id and support.session_id in excluded_sessions) + ): + changed = True + continue + if sanitized is not support: + changed = True + filtered_results.append(sanitized) + if not changed: return pack return pack.model_copy(update={"results": filtered_results}) +def _filter_pack_for_snapshot( + pack: AssistantProofPack | None, + snapshot: AssistantTargetSnapshot, +) -> AssistantProofPack | None: + """Apply the requesting workflow's stable run/session exclusions to reused packs.""" + excluded_ids = _current_run_ids(snapshot) + return _drop_current_run_supports_from_pack( + pack, + excluded_run_ids=excluded_ids, + excluded_session_ids=excluded_ids, + ) + + def _is_current_run_record(record: UnifiedProofSearchRecord) -> bool: if record.corpus == "syntheticlib4": return False @@ -884,6 +1374,14 @@ def _is_current_run_support(support: AssistantProofSupport) -> bool: return len(parts) >= 3 and parts[1] == active_session_id +def _current_run_ids(snapshot: AssistantTargetSnapshot) -> list[str]: + return sorted({ + value.strip() + for value in (snapshot.run_id, _active_autonomous_session_id()) + if value and value.strip() + }) + + def _cooldown_delay_for_stage(stage: int) -> int: if stage <= 0: return 0 @@ -1140,6 +1638,19 @@ def _write_json(path: Path, payload: dict[str, Any]) -> None: path.write_text(json.dumps(payload, ensure_ascii=True, indent=2), encoding="utf-8") +def _read_latest_assistant_pack() -> AssistantProofPack | None: + path = _assistant_pack_path() + if not path.exists(): + return None + try: + return _drop_current_run_supports_from_pack( + AssistantProofPack.model_validate_json(path.read_text(encoding="utf-8")) + ) + except Exception: + logger.debug("Assistant latest-pack file could not be read", exc_info=True) + return None + + def _delete_if_exists(path: Path) -> None: try: path.unlink(missing_ok=True) diff --git a/backend/shared/proof_search/assistant_models.py b/backend/shared/proof_search/assistant_models.py index 99d5dd2..41fff83 100644 --- a/backend/shared/proof_search/assistant_models.py +++ b/backend/shared/proof_search/assistant_models.py @@ -45,6 +45,7 @@ "cached_oauth_cooldown", "deterministic_oauth_cooldown", ] +ASSISTANT_PROOF_PACK_SCHEMA_VERSION = "moto.assistant_proof_pack.v2" def _now_iso() -> str: @@ -77,8 +78,9 @@ class AssistantTargetSnapshot(BaseModel): source_title: str = "" source_type: str = "" source_id: str = "" + run_id: str = "" dependency_names: list[str] = Field(default_factory=list) - imports: list[str] = Field(default_factory=lambda: ["Mathlib"]) + imports: list[str] = Field(default_factory=list) target_hash: str = "" created_at: str = Field(default_factory=_now_iso) @@ -107,6 +109,7 @@ def stable_hash(self) -> str: self.source_title, self.source_type, self.source_id, + self.run_id, " ".join(self.source_titles), " ".join(self.dependency_names), " ".join(self.imports), @@ -148,6 +151,7 @@ class AssistantProofSupport(BaseModel): source_kind: str = "verified_proof" proof_id: str session_id: str = "" + run_id: str = "" fingerprint: str = "" theorem_name: str = "" theorem_statement: str @@ -157,10 +161,20 @@ class AssistantProofSupport(BaseModel): theorem_statement_hash: str = "" lean_code_hash: str = "" canonical_uri: str = "" + source_type: str = "" + source_id: str = "" + source_title: str = "" + novelty_tier: str = "" + novelty_reasoning: str = "" + created_at: str = "" relevance_reason: str = "" transfer_hint: str = "" has_hydrated_code: bool = False lean_code: str = "" + retrieval_lanes: list[str] = Field(default_factory=list) + occurrence_provenance: list[dict[str, str]] = Field(default_factory=list) + occurrence_total: int = 0 + occurrence_omitted: int = 0 @classmethod def from_record( @@ -170,6 +184,24 @@ def from_record( relevance_reason: str = "", transfer_hint: str = "", ) -> "AssistantProofSupport": + occurrences = record.metadata.get("assistant_occurrences") + if not isinstance(occurrences, list): + occurrence = { + "search_id": record.search_id, + "corpus": record.corpus, + "corpus_scope": record.corpus_scope or record.release_id, + "session_id": record.session_id, + "run_id": record.run_id, + "source_type": record.source_type, + "source_id": record.source_id, + "source_title": record.source_title, + } + if record.metadata.get("assistant_exclude_standalone_exact_duplicate_emphasis") is True: + occurrence["assistant_exclude_standalone_exact_duplicate_emphasis"] = "true" + occurrences = [occurrence] + lanes = record.metadata.get("assistant_retrieval_lanes") + if not isinstance(lanes, list): + lanes = [] return cls( search_id=record.search_id, corpus=record.corpus, @@ -177,6 +209,7 @@ def from_record( source_kind=record.source_kind, proof_id=record.proof_id, session_id=record.session_id, + run_id=record.run_id, fingerprint=record.external_fingerprint, theorem_name=record.theorem_name or record.display_title, theorem_statement=record.theorem_statement, @@ -186,10 +219,24 @@ def from_record( theorem_statement_hash=record.theorem_statement_hash, lean_code_hash=record.lean_code_hash, canonical_uri=record.canonical_uri, + source_type=record.source_type, + source_id=record.source_id, + source_title=record.source_title, + novelty_tier=record.novelty_tier, + novelty_reasoning=record.novelty_reasoning, + created_at=record.created_at, relevance_reason=relevance_reason, transfer_hint=transfer_hint, has_hydrated_code=bool((record.lean_code or "").strip()), lean_code=record.lean_code or "", + retrieval_lanes=[str(value) for value in lanes if str(value)], + occurrence_provenance=[ + {str(key): str(value) for key, value in occurrence.items()} + for occurrence in occurrences + if isinstance(occurrence, dict) + ], + occurrence_total=int(record.metadata.get("assistant_occurrence_total") or len(occurrences)), + occurrence_omitted=int(record.metadata.get("assistant_occurrence_omitted") or 0), ) def metadata_only_dump(self) -> dict: @@ -198,11 +245,44 @@ def metadata_only_dump(self) -> dict: payload["has_hydrated_code"] = bool(self.has_hydrated_code) return payload + def event_preview_dump(self, *, occurrence_limit: int = 8) -> dict: + payload = self.metadata_only_dump() + occurrences = payload.get("occurrence_provenance", []) + payload["occurrence_provenance"] = occurrences[:occurrence_limit] + payload["occurrence_total"] = max(self.occurrence_total, len(occurrences)) + payload["occurrence_omitted"] = max(0, self.occurrence_total - len(payload["occurrence_provenance"])) + payload["occurrence_detail_available"] = self.occurrence_total > len(payload["occurrence_provenance"]) + return payload + + +class AssistantSupportLineageOccurrence(BaseModel): + """Metadata-only provenance for one occurrence of an Assistant support.""" + + search_id: str = "" + corpus: str = "" + corpus_scope: str = "" + session_id: str = "" + run_id: str = "" + source_type: str = "" + source_id: str = "" + source_title: str = "" + + +class AssistantSupportLineageResponse(BaseModel): + """Bounded page of provenance for one support in one Assistant target pack.""" + + target_hash: str + support_search_id: str + occurrence_total: int + offset: int + limit: int + next_offset: int | None = None + occurrences: list[AssistantSupportLineageOccurrence] = Field(default_factory=list) class AssistantProofPack(BaseModel): """Latest non-blocking proof-support pack for one target snapshot.""" - schema_version: str = "moto.assistant_proof_pack.v1" + schema_version: str = ASSISTANT_PROOF_PACK_SCHEMA_VERSION created_at: str = Field(default_factory=_now_iso) workflow_mode: AssistantWorkflowMode target_kind: AssistantTargetKind @@ -217,6 +297,7 @@ class AssistantProofPack(BaseModel): candidate_count: int = 0 shortlist_count: int = 0 selection_reasoning: str = "" + retrieval_observability: dict = Field(default_factory=dict) def to_prompt_context(self, *, max_code_chars_per_result: int = 4000) -> str: return self._to_prompt_context( @@ -248,8 +329,12 @@ def _to_prompt_context( f"Selection mode: {self.selection_mode}", f"Query summary: {self.query_summary or '[not provided]'}", ( - "Use these verified memory records only as relevant mathematical context, " - "proof-pattern, dependency, or tactic guidance for the user's prompt/current target." + "These are Lean-verified theorem records. Use them only when their exact statement " + "or proof structure materially helps the user's objective/current task. They may " + "provide mathematical context, bounds, invariants, impossibility arguments, " + "decomposition patterns, dependencies, or tactics. They do not override or " + "mathematically reinterpret the user objective, prove informal applicability, or " + "require the current work to become mathematical. Ignore unrelated records." ), ] for index, support in enumerate(self.results[:7], start=1): @@ -289,3 +374,27 @@ def metadata_only_dump(self) -> dict: payload["results"] = [result.metadata_only_dump() for result in self.results] return payload + +class LatestAssistantProofPackResponse(BaseModel): + """Metadata-only latest Assistant pack exposed to UI clients.""" + + enabled: bool + has_pack: bool + results: list[dict] = Field(default_factory=list) + disabled_reason: str = "" + schema_version: str = "" + created_at: str = "" + workflow_mode: str = "" + target_kind: str = "" + target_hash: str = "" + query_summary: str = "" + freshness: str = "" + warnings: list[str] = Field(default_factory=list) + selection_mode: str = "" + assistant_role_id: str = "" + assistant_model_id: str = "" + candidate_count: int = 0 + shortlist_count: int = 0 + selection_reasoning: str = "" + retrieval_observability: dict = Field(default_factory=dict) + diff --git a/backend/shared/proof_search/assistant_ranker.py b/backend/shared/proof_search/assistant_ranker.py index 0017d54..ed5a9c7 100644 --- a/backend/shared/proof_search/assistant_ranker.py +++ b/backend/shared/proof_search/assistant_ranker.py @@ -20,6 +20,7 @@ "leanoj": 0.95, "syntheticlib4": 0.9, } +_MIN_TARGET_RELEVANCE = 0.015 @dataclass(frozen=True) @@ -189,6 +190,8 @@ def _rank_candidates( + 0.10 * lexical_score + 0.05 * recency_score ) + if max(lexical_score, dependency_score, import_score, exact_score) < _MIN_TARGET_RELEVANCE: + continue ranked.append( RankedProofCandidate( record=record, @@ -209,7 +212,14 @@ def _rank_candidates( ) ) - ranked.sort(key=lambda candidate: candidate.score, reverse=True) + ranked.sort( + key=lambda candidate: ( + -candidate.score, + -candidate.exact_score, + -candidate.lexical_score, + candidate.record.search_id, + ) + ) return ranked diff --git a/backend/shared/proof_search/indexer.py b/backend/shared/proof_search/indexer.py index 71d54a2..730a8be 100644 --- a/backend/shared/proof_search/indexer.py +++ b/backend/shared/proof_search/indexer.py @@ -2,13 +2,16 @@ from __future__ import annotations import json +import os import re import sqlite3 +import uuid from collections import Counter from contextlib import closing from pathlib import Path from typing import Any, Iterable +from backend.shared.proof_identity import CANONICAL_PROOF_IDENTITY_VERSION from backend.shared.proof_search.models import ( CorpusOverview, ProofSearchRequest, @@ -17,6 +20,7 @@ ) RESULT_CAP = 7 +PROOF_SEARCH_INDEX_SCHEMA_VERSION = 3 class ProofSearchIndexer: @@ -26,22 +30,71 @@ def __init__(self, db_path: Path) -> None: self.db_path = Path(db_path) def rebuild(self, records: Iterable[UnifiedProofSearchRecord]) -> None: + """Build a complete sibling index and atomically publish it.""" self.db_path.parent.mkdir(parents=True, exist_ok=True) unique_records = {record.search_id: record for record in records} + temporary_path = self.db_path.with_name( + f".{self.db_path.name}.rebuild-{uuid.uuid4().hex}" + ) + temporary_indexer = ProofSearchIndexer(temporary_path) + try: + temporary_indexer._populate_new_index(unique_records.values()) + if not temporary_indexer.is_compatible(): + raise RuntimeError("Rebuilt proof-search index failed compatibility validation") + os.replace(temporary_path, self.db_path) + finally: + for candidate in ( + temporary_path, + Path(f"{temporary_path}-wal"), + Path(f"{temporary_path}-shm"), + ): + try: + candidate.unlink() + except FileNotFoundError: + pass + + def _populate_new_index( + self, + records: Iterable[UnifiedProofSearchRecord], + ) -> None: + """Populate a newly created derived index before publication.""" + records = tuple(records) with closing(self._connect()) as conn: self._create_schema(conn) - conn.execute("DELETE FROM proof_fts") - conn.execute("DELETE FROM proof_records") conn.executemany( _PROOF_RECORD_INSERT_SQL, - (self._record_payload(record) for record in unique_records.values()), + (self._record_payload(record) for record in records), ) conn.executemany( _PROOF_FTS_INSERT_SQL, - (self._fts_payload(record) for record in unique_records.values()), + (self._fts_payload(record) for record in records), + ) + conn.execute(f"PRAGMA user_version = {PROOF_SEARCH_INDEX_SCHEMA_VERSION}") + conn.execute( + "INSERT OR REPLACE INTO proof_index_metadata(key, value) VALUES (?, ?)", + ("canonical_identity_version", CANONICAL_PROOF_IDENTITY_VERSION), ) conn.commit() + def is_compatible(self) -> bool: + """Return whether this derived index matches current schema/identity versions.""" + if not self.db_path.exists(): + return False + try: + with closing(self._connect()) as conn: + schema_version = int(conn.execute("PRAGMA user_version").fetchone()[0]) + row = conn.execute( + "SELECT value FROM proof_index_metadata WHERE key = ?", + ("canonical_identity_version",), + ).fetchone() + return ( + schema_version == PROOF_SEARCH_INDEX_SCHEMA_VERSION + and row is not None + and str(row["value"]) == CANONICAL_PROOF_IDENTITY_VERSION + ) + except (sqlite3.Error, TypeError, ValueError): + return False + def search(self, request: ProofSearchRequest) -> ProofSearchResponse: if not self.db_path.exists(): return ProofSearchResponse( @@ -86,6 +139,7 @@ def search_candidate_pool( pool_limit: int, exclude_corpus_scopes: Iterable[str] | None = None, exclude_session_ids: Iterable[str] | None = None, + exclude_run_ids: Iterable[str] | None = None, ) -> list[UnifiedProofSearchRecord]: """Return a wider internal candidate pool without changing public route caps.""" if not self.db_path.exists(): @@ -99,6 +153,7 @@ def search_candidate_pool( candidate_limit=pool_limit * 4, exclude_corpus_scopes=exclude_corpus_scopes, exclude_session_ids=exclude_session_ids, + exclude_run_ids=exclude_run_ids, ) records = self._dedupe_and_limit(candidates, request, result_cap=pool_limit) @@ -106,12 +161,158 @@ def search_candidate_pool( records = [record.model_copy(update={"lean_code": ""}) for record in records] return records + def exact_identity_neighborhood( + self, + *, + theorem_statement_hashes: Iterable[str], + lean_code_hashes: Iterable[str], + corpora: Iterable[str], + exclude_run_ids: Iterable[str] | None = None, + exclude_session_ids: Iterable[str] | None = None, + identity_version: str = CANONICAL_PROOF_IDENTITY_VERSION, + limit: int = 256, + ) -> list[UnifiedProofSearchRecord]: + """Resolve exact artifacts to their exact occurrence neighborhoods. + + Occurrence identity is composite. Never independently union run, + session, and source columns: values can be reused by unrelated records. + """ + if not self.db_path.exists(): + return [] + statement_hashes = sorted({value for value in theorem_statement_hashes if value}) + code_hashes = sorted({value for value in lean_code_hashes if value}) + corpus_values = sorted({value for value in corpora if value}) + if not (statement_hashes or code_hashes) or not corpus_values: + return [] + excluded_runs = sorted({value for value in (exclude_run_ids or []) if value}) + excluded_sessions = sorted({value for value in (exclude_session_ids or []) if value}) + identity_predicates: list[str] = [] + if statement_hashes: + identity_predicates.append( + "EXISTS (SELECT 1 FROM requested_statement_hashes requested " + "WHERE requested.value = records.canonical_theorem_statement_hash)" + ) + if code_hashes: + identity_predicates.append( + "EXISTS (SELECT 1 FROM requested_code_hashes requested " + "WHERE requested.value = records.canonical_lean_code_hash)" + ) + exact_identity_sql = " AND ".join(identity_predicates) + with closing(self._connect()) as conn: + self._create_schema(conn) + conn.executescript( + """ + CREATE TEMP TABLE requested_statement_hashes ( + value TEXT PRIMARY KEY + ) WITHOUT ROWID; + CREATE TEMP TABLE requested_code_hashes ( + value TEXT PRIMARY KEY + ) WITHOUT ROWID; + CREATE TEMP TABLE requested_corpora ( + value TEXT PRIMARY KEY + ) WITHOUT ROWID; + CREATE TEMP TABLE excluded_neighborhood_runs ( + value TEXT PRIMARY KEY + ) WITHOUT ROWID; + CREATE TEMP TABLE excluded_neighborhood_sessions ( + value TEXT PRIMARY KEY + ) WITHOUT ROWID; + CREATE TEMP TABLE exact_occurrence_keys ( + corpus TEXT NOT NULL, + run_id TEXT NOT NULL, + session_id TEXT NOT NULL, + source_type TEXT NOT NULL, + source_id TEXT NOT NULL, + PRIMARY KEY (corpus, run_id, session_id, source_type, source_id) + ) WITHOUT ROWID; + """ + ) + conn.executemany( + "INSERT INTO requested_statement_hashes(value) VALUES (?)", + ((value,) for value in statement_hashes), + ) + conn.executemany( + "INSERT INTO requested_code_hashes(value) VALUES (?)", + ((value,) for value in code_hashes), + ) + conn.executemany( + "INSERT INTO requested_corpora(value) VALUES (?)", + ((value,) for value in corpus_values), + ) + conn.executemany( + "INSERT INTO excluded_neighborhood_runs(value) VALUES (?)", + ((value,) for value in excluded_runs), + ) + conn.executemany( + "INSERT INTO excluded_neighborhood_sessions(value) VALUES (?)", + ((value,) for value in excluded_sessions), + ) + conn.execute( + f""" + INSERT INTO exact_occurrence_keys ( + corpus, run_id, session_id, source_type, source_id + ) + SELECT DISTINCT + records.corpus, + COALESCE(records.run_id, ''), + COALESCE(records.session_id, ''), + COALESCE(records.source_type, ''), + COALESCE(records.source_id, '') + FROM proof_records AS records + INNER JOIN requested_corpora AS requested_corpus + ON requested_corpus.value = records.corpus + WHERE {exact_identity_sql} + AND records.canonical_identity_version = ? + AND NOT EXISTS ( + SELECT 1 FROM excluded_neighborhood_runs excluded + WHERE excluded.value = COALESCE(records.run_id, '') + ) + AND NOT EXISTS ( + SELECT 1 FROM excluded_neighborhood_sessions excluded + WHERE excluded.value = COALESCE(records.session_id, '') + ) + """, + (identity_version,), + ) + if conn.execute("SELECT 1 FROM exact_occurrence_keys LIMIT 1").fetchone() is None: + return [] + rows = conn.execute( + f""" + SELECT records.* + FROM proof_records AS records + INNER JOIN exact_occurrence_keys AS occurrence + ON occurrence.corpus = records.corpus + AND occurrence.run_id = COALESCE(records.run_id, '') + AND occurrence.session_id = COALESCE(records.session_id, '') + AND occurrence.source_type = COALESCE(records.source_type, '') + AND occurrence.source_id = COALESCE(records.source_id, '') + WHERE NOT EXISTS ( + SELECT 1 FROM excluded_neighborhood_runs excluded + WHERE excluded.value = COALESCE(records.run_id, '') + ) + AND NOT EXISTS ( + SELECT 1 FROM excluded_neighborhood_sessions excluded + WHERE excluded.value = COALESCE(records.session_id, '') + ) + ORDER BY + CASE WHEN {exact_identity_sql} + AND records.canonical_identity_version = ? THEN 0 ELSE 1 END, + records.created_at DESC, + records.search_id ASC + LIMIT ? + """, + (identity_version, max(1, limit)), + ).fetchall() + return [self._row_to_record(row) for row in rows] + def get_record( self, *, corpus: str, proof_id: str, session_id: str | None = None, + search_id: str | None = None, + run_id: str | None = None, ) -> UnifiedProofSearchRecord | None: """Fetch one indexed record for detail/hydration flows.""" if not self.db_path.exists(): @@ -119,11 +320,20 @@ def get_record( with closing(self._connect()) as conn: self._create_schema(conn) - clauses = ["corpus = ?", "(proof_id = ? OR search_id = ?)"] - params: list[Any] = [corpus, proof_id, proof_id] + clauses = ["corpus = ?"] + params: list[Any] = [corpus] + if search_id: + clauses.append("search_id = ?") + params.append(search_id) + else: + clauses.append("(proof_id = ? OR search_id = ?)") + params.extend([proof_id, proof_id]) if session_id: clauses.append("session_id = ?") params.append(session_id) + if run_id: + clauses.append("run_id = ?") + params.append(run_id) row = conn.execute( f""" SELECT * @@ -143,6 +353,51 @@ def get_record( return self._row_to_record(row) if row else None + def support_lineage( + self, + *, + theorem_statement_hash: str, + lean_code_hash: str, + corpora: Iterable[str], + exclude_run_ids: Iterable[str] | None, + exclude_session_ids: Iterable[str] | None, + offset: int, + limit: int, + ) -> tuple[int, list[dict[str, str]]]: + """Return an authoritative metadata-only page for an exact artifact.""" + corpus_values = sorted({value for value in corpora if value}) + identities = [ + ("theorem_statement_hash = ?", theorem_statement_hash), + ("lean_code_hash = ?", lean_code_hash), + ] + identities = [(clause, value) for clause, value in identities if value] + if not self.db_path.exists() or not corpus_values or not identities: + return 0, [] + clauses = [ + f"({' AND '.join(clause for clause, _ in identities)})", + f"corpus IN ({','.join('?' for _ in corpus_values)})", + ] + params: list[Any] = [value for _, value in identities] + corpus_values + for column, values in ( + ("run_id", sorted({v for v in (exclude_run_ids or []) if v})), + ("session_id", sorted({v for v in (exclude_session_ids or []) if v})), + ): + if values: + clauses.append(f"COALESCE({column}, '') NOT IN ({','.join('?' for _ in values)})") + params.extend(values) + where = " AND ".join(clauses) + with closing(self._connect()) as conn: + self._create_schema(conn) + total = int(conn.execute(f"SELECT COUNT(*) FROM proof_records WHERE {where}", params).fetchone()[0]) + rows = conn.execute( + f"""SELECT search_id, corpus, corpus_scope, session_id, run_id, + source_type, source_id, source_title + FROM proof_records WHERE {where} + ORDER BY created_at DESC, search_id ASC LIMIT ? OFFSET ?""", + [*params, limit, offset], + ).fetchall() + return total, [{key: str(row[key] or "") for key in row.keys()} for row in rows] + def overview(self, corpora: Iterable[str] | None = None) -> CorpusOverview: if not self.db_path.exists(): return CorpusOverview( @@ -231,6 +486,7 @@ def _create_schema(self, conn: sqlite3.Connection) -> None: external_fingerprint TEXT NOT NULL, release_id TEXT NOT NULL, session_id TEXT NOT NULL, + run_id TEXT NOT NULL DEFAULT '', source_type TEXT NOT NULL, source_id TEXT NOT NULL, source_title TEXT NOT NULL, @@ -243,6 +499,9 @@ def _create_schema(self, conn: sqlite3.Connection) -> None: lean_code TEXT NOT NULL, lean_code_hash TEXT NOT NULL, theorem_statement_hash TEXT NOT NULL, + canonical_identity_version TEXT NOT NULL DEFAULT '', + canonical_lean_code_hash TEXT NOT NULL DEFAULT '', + canonical_theorem_statement_hash TEXT NOT NULL DEFAULT '', imports_json TEXT NOT NULL, dependency_names_json TEXT NOT NULL, topic_tags_json TEXT NOT NULL, @@ -258,6 +517,25 @@ def _create_schema(self, conn: sqlite3.Connection) -> None: ) """ ) + columns = {str(row["name"]) for row in conn.execute("PRAGMA table_info(proof_records)")} + if "run_id" not in columns: + conn.execute("ALTER TABLE proof_records ADD COLUMN run_id TEXT NOT NULL DEFAULT ''") + if "canonical_identity_version" not in columns: + conn.execute( + "ALTER TABLE proof_records ADD COLUMN canonical_identity_version TEXT NOT NULL DEFAULT ''" + ) + if "canonical_lean_code_hash" not in columns: + conn.execute( + "ALTER TABLE proof_records ADD COLUMN canonical_lean_code_hash TEXT NOT NULL DEFAULT ''" + ) + if "canonical_theorem_statement_hash" not in columns: + conn.execute( + "ALTER TABLE proof_records ADD COLUMN canonical_theorem_statement_hash TEXT NOT NULL DEFAULT ''" + ) + conn.execute( + "CREATE TABLE IF NOT EXISTS proof_index_metadata " + "(key TEXT PRIMARY KEY, value TEXT NOT NULL)" + ) conn.execute( """ CREATE VIRTUAL TABLE IF NOT EXISTS proof_fts USING fts5( @@ -328,6 +606,7 @@ def _query_candidates( candidate_limit: int | None = None, exclude_corpus_scopes: Iterable[str] | None = None, exclude_session_ids: Iterable[str] | None = None, + exclude_run_ids: Iterable[str] | None = None, ) -> list[tuple[UnifiedProofSearchRecord, float]]: clauses = [] params: list[Any] = [] @@ -370,6 +649,11 @@ def _query_candidates( placeholders = ",".join("?" for _ in excluded_sessions) clauses.append(f"r.session_id NOT IN ({placeholders})") params.extend(excluded_sessions) + excluded_runs = [run_id for run_id in (exclude_run_ids or []) if run_id] + if excluded_runs: + placeholders = ",".join("?" for _ in excluded_runs) + clauses.append(f"r.run_id NOT IN ({placeholders})") + params.extend(excluded_runs) where_sql = f"WHERE {' AND '.join(clauses)}" if clauses else "" fts_query = _build_fts_query( @@ -390,12 +674,12 @@ def _query_candidates( FROM proof_fts JOIN proof_records r ON r.search_id = proof_fts.search_id {where_sql} {'AND' if where_sql else 'WHERE'} proof_fts MATCH ? - ORDER BY rank ASC + ORDER BY rank ASC, r.search_id ASC LIMIT ? """ rows = conn.execute(sql, [*params, fts_query, limit]).fetchall() else: - sql = f"SELECT r.*, 0.0 AS rank FROM proof_records r {where_sql} LIMIT ?" + sql = f"SELECT r.*, 0.0 AS rank FROM proof_records r {where_sql} ORDER BY r.search_id ASC LIMIT ?" rows = conn.execute(sql, [*params, limit]).fetchall() candidates: list[tuple[UnifiedProofSearchRecord, float]] = [] @@ -405,7 +689,7 @@ def _query_candidates( score += self._exact_boost(record, request) candidates.append((record, score)) - candidates.sort(key=lambda item: item[1], reverse=True) + candidates.sort(key=lambda item: (-item[1], item[0].search_id)) return candidates def _dedupe_and_limit( @@ -467,6 +751,7 @@ def _json_list(field: str) -> list[str]: external_fingerprint=row["external_fingerprint"], release_id=row["release_id"], session_id=row["session_id"], + run_id=row["run_id"], source_type=row["source_type"], source_id=row["source_id"], source_title=row["source_title"], @@ -479,6 +764,15 @@ def _json_list(field: str) -> list[str]: lean_code=row["lean_code"], lean_code_hash=row["lean_code_hash"], theorem_statement_hash=row["theorem_statement_hash"], + canonical_identity_version=str( + row["canonical_identity_version"] or "" + ), + canonical_lean_code_hash=str( + row["canonical_lean_code_hash"] or "" + ), + canonical_theorem_statement_hash=str( + row["canonical_theorem_statement_hash"] or "" + ), imports=_json_list("imports_json"), dependency_names=_json_list("dependency_names_json"), topic_tags=_json_list("topic_tags_json"), @@ -542,19 +836,21 @@ def _top_counts(counter: Counter[str], limit: int = 10) -> list[dict[str, Any]]: _PROOF_RECORD_INSERT_SQL = """ INSERT OR REPLACE INTO proof_records ( search_id, corpus, corpus_scope, source_kind, proof_id, - external_fingerprint, release_id, session_id, source_type, source_id, + external_fingerprint, release_id, session_id, run_id, source_type, source_id, source_title, display_title, theorem_name, theorem_statement, informal_statement, proof_description, formal_sketch, lean_code, - lean_code_hash, theorem_statement_hash, imports_json, + lean_code_hash, theorem_statement_hash, canonical_identity_version, + canonical_lean_code_hash, canonical_theorem_statement_hash, imports_json, dependency_names_json, topic_tags_json, domain_tags_json, module, source_path, novelty_tier, novelty_reasoning, verified, created_at, canonical_uri, metadata_json ) VALUES ( :search_id, :corpus, :corpus_scope, :source_kind, :proof_id, - :external_fingerprint, :release_id, :session_id, :source_type, :source_id, + :external_fingerprint, :release_id, :session_id, :run_id, :source_type, :source_id, :source_title, :display_title, :theorem_name, :theorem_statement, :informal_statement, :proof_description, :formal_sketch, :lean_code, - :lean_code_hash, :theorem_statement_hash, :imports_json, + :lean_code_hash, :theorem_statement_hash, :canonical_identity_version, + :canonical_lean_code_hash, :canonical_theorem_statement_hash, :imports_json, :dependency_names_json, :topic_tags_json, :domain_tags_json, :module, :source_path, :novelty_tier, :novelty_reasoning, :verified, :created_at, :canonical_uri, :metadata_json diff --git a/backend/shared/proof_search/models.py b/backend/shared/proof_search/models.py index 3e24c94..da27165 100644 --- a/backend/shared/proof_search/models.py +++ b/backend/shared/proof_search/models.py @@ -33,6 +33,7 @@ class UnifiedProofSearchRecord(BaseModel): external_fingerprint: str = "" release_id: str = "" session_id: str = "" + run_id: str = "" source_type: str = "" source_id: str = "" source_title: str = "" @@ -45,6 +46,9 @@ class UnifiedProofSearchRecord(BaseModel): lean_code: str = "" lean_code_hash: str = "" theorem_statement_hash: str = "" + canonical_identity_version: str = "" + canonical_lean_code_hash: str = "" + canonical_theorem_statement_hash: str = "" imports: list[str] = Field(default_factory=list) dependency_names: list[str] = Field(default_factory=list) topic_tags: list[str] = Field(default_factory=list) @@ -65,8 +69,11 @@ def dedupe_keys(self) -> list[str]: keys.append(f"fingerprint:{self.external_fingerprint}") if self.corpus != "syntheticlib4" and self.proof_id: keys.append(f"local:{self.corpus}:{self.proof_id}") - if self.theorem_statement_hash and self.lean_code_hash: - keys.append(f"hash:{self.theorem_statement_hash}:{self.lean_code_hash}") + statement_hash = self.canonical_theorem_statement_hash or self.theorem_statement_hash + lean_hash = self.canonical_lean_code_hash or self.lean_code_hash + identity_version = self.canonical_identity_version or "external-integrity" + if statement_hash and lean_hash: + keys.append(f"hash:{identity_version}:{statement_hash}:{lean_hash}") return keys diff --git a/backend/shared/proof_search/moto_sources.py b/backend/shared/proof_search/moto_sources.py index 28580f8..23747b4 100644 --- a/backend/shared/proof_search/moto_sources.py +++ b/backend/shared/proof_search/moto_sources.py @@ -1,14 +1,21 @@ """Canonical MOTO proof database normalization for unified proof search.""" from __future__ import annotations -import hashlib from pathlib import Path from typing import Any from backend.autonomous.memory.proof_database import manual_proof_database, proof_database from backend.shared.models import ProofDependency, ProofRecord +from backend.shared.proof_identity import canonical_proof_identity from backend.shared.proof_search.models import UnifiedProofSearchRecord +ASSISTANT_EXCLUDE_STANDALONE_EXACT_DUPLICATE_EMPHASIS = ( + "assistant_exclude_standalone_exact_duplicate_emphasis" +) +STANDALONE_EXACT_DUPLICATE_EMPHASIS_PURPOSE = ( + "standalone_exact_duplicate_emphasis" +) + async def load_moto_proof_records() -> list[UnifiedProofSearchRecord]: """Collect current canonical MOTO proof records without reading display appendices.""" @@ -53,17 +60,36 @@ async def _records_from_autonomous_history() -> list[UnifiedProofSearchRecord]: async def _records_from_manual_history() -> list[UnifiedProofSearchRecord]: + history_root: Path try: from backend.shared.config import system_config + history_root = Path(system_config.data_dir) / "manual_proof_runs" entries = await manual_proof_database.list_proof_library_from_history( - Path(system_config.data_dir) / "manual_proof_runs", + history_root, novel_only=False, ) except Exception: return [] - return [_record_from_library_entry(entry, default_corpus="manual") for entry in entries] + records: list[UnifiedProofSearchRecord] = [] + for entry in entries: + full_entry = entry + session_id = str(entry.get("session_id", "") or "") + proof_id = str(entry.get("proof_id", "") or "") + if session_id and proof_id: + try: + hydrated = await manual_proof_database.get_library_proof_from_history( + history_root, + session_id, + proof_id, + ) + except Exception: + hydrated = None + if hydrated: + full_entry = {**entry, **hydrated} + records.append(_record_from_library_entry(full_entry, default_corpus="manual")) + return records def _record_from_library_entry( @@ -74,9 +100,11 @@ def _record_from_library_entry( theorem_statement = str(entry.get("theorem_statement", "") or "") proof_id = str(entry.get("proof_id", "") or "") session_id = str(entry.get("session_id", "") or "") + run_id = str(entry.get("run_id", "") or session_id) source_type = str(entry.get("source_type", "") or "") corpus = "leanoj" if source_type.startswith("leanoj_") else default_corpus lean_code = str(entry.get("lean_code", "") or "") + identity = canonical_proof_identity(theorem_statement, lean_code) return UnifiedProofSearchRecord( search_id=f"{corpus}:{session_id}:{proof_id}", corpus=corpus, @@ -84,6 +112,7 @@ def _record_from_library_entry( source_kind="verified_proof", proof_id=proof_id, session_id=session_id, + run_id=run_id, source_type=source_type, source_id=str(entry.get("source_id", "") or ""), source_title=str(entry.get("source_title", "") or ""), @@ -92,8 +121,11 @@ def _record_from_library_entry( theorem_statement=theorem_statement, formal_sketch=str(entry.get("formal_sketch", "") or ""), lean_code=lean_code, - lean_code_hash=_sha256_text(lean_code) if lean_code else "", - theorem_statement_hash=_sha256_text(theorem_statement), + lean_code_hash=identity.lean_code_hash if lean_code else "", + theorem_statement_hash=identity.theorem_statement_hash, + canonical_identity_version=identity.version, + canonical_lean_code_hash=identity.lean_code_hash if lean_code else "", + canonical_theorem_statement_hash=identity.theorem_statement_hash, dependency_names=_dependency_names(entry.get("dependencies", [])), novelty_tier=str(entry.get("novelty_tier", "") or ""), novelty_reasoning=str(entry.get("novelty_reasoning", "") or ""), @@ -106,6 +138,14 @@ def _record_from_library_entry( "attempt_count": entry.get("attempt_count", 0), "verification_notes": str(entry.get("verification_notes", "") or ""), "user_prompt": str(entry.get("user_prompt", "") or ""), + "run_id": run_id, + "canonical_identity_version": identity.version, + "canonical_theorem_statement_hash": identity.theorem_statement_hash, + "canonical_lean_code_hash": identity.lean_code_hash if lean_code else "", + ASSISTANT_EXCLUDE_STANDALONE_EXACT_DUPLICATE_EMPHASIS: ( + entry.get("artifact_purpose") + == STANDALONE_EXACT_DUPLICATE_EMPHASIS_PURPOSE + ), }, ) @@ -118,9 +158,9 @@ def normalize_proof_record( ) -> UnifiedProofSearchRecord: """Convert a stored ProofRecord into the shared search model.""" corpus = "leanoj" if proof.source_type.startswith("leanoj_") else default_corpus - lean_code_hash = _sha256_text(proof.lean_code) if proof.lean_code else "" - statement_hash = _sha256_text(proof.theorem_statement) + identity = canonical_proof_identity(proof.theorem_statement, proof.lean_code) scope = "active" if default_corpus == "manual" else "current" + run_id = str(getattr(proof, "run_id", "") or session_id) return UnifiedProofSearchRecord( search_id=f"{corpus}:{session_id}:{proof.proof_id}", @@ -129,6 +169,7 @@ def normalize_proof_record( source_kind="verified_proof", proof_id=proof.proof_id, session_id=session_id, + run_id=run_id, source_type=proof.source_type, source_id=proof.source_id, source_title=proof.source_title, @@ -137,8 +178,11 @@ def normalize_proof_record( theorem_statement=proof.theorem_statement, formal_sketch=proof.formal_sketch, lean_code=proof.lean_code, - lean_code_hash=lean_code_hash, - theorem_statement_hash=statement_hash, + lean_code_hash=identity.lean_code_hash if proof.lean_code else "", + theorem_statement_hash=identity.theorem_statement_hash, + canonical_identity_version=identity.version, + canonical_lean_code_hash=identity.lean_code_hash if proof.lean_code else "", + canonical_theorem_statement_hash=identity.theorem_statement_hash, imports=_import_names(proof.dependencies), dependency_names=_dependency_names(proof.dependencies), novelty_tier=proof.novelty_tier, @@ -152,6 +196,13 @@ def normalize_proof_record( "novel": proof.novel, "verification_notes": proof.verification_notes, "attempt_count": proof.attempt_count, + ASSISTANT_EXCLUDE_STANDALONE_EXACT_DUPLICATE_EMPHASIS: ( + proof.artifact_purpose + == STANDALONE_EXACT_DUPLICATE_EMPHASIS_PURPOSE + ), + "canonical_identity_version": identity.version, + "canonical_theorem_statement_hash": identity.theorem_statement_hash, + "canonical_lean_code_hash": identity.lean_code_hash if proof.lean_code else "", }, ) @@ -179,7 +230,3 @@ def _import_names(dependencies: list[Any]) -> list[str]: imports.append(str(name).split(".")[0]) return sorted(set(imports)) - -def _sha256_text(value: str) -> str: - return hashlib.sha256((value or "").encode("utf-8")).hexdigest() - diff --git a/backend/shared/proof_search/search_service.py b/backend/shared/proof_search/search_service.py index d3455dc..574e538 100644 --- a/backend/shared/proof_search/search_service.py +++ b/backend/shared/proof_search/search_service.py @@ -6,6 +6,7 @@ from backend.shared.config import system_config from backend.shared.proof_search.indexer import ProofSearchIndexer +from backend.shared.proof_identity import CANONICAL_PROOF_IDENTITY_VERSION from backend.shared.proof_search.models import ( CorpusOverview, ProofSearchRequest, @@ -18,14 +19,20 @@ load_syntheticlib4_fixture_records, normalize_syntheticlib4_record, ) -from backend.shared.syntheticlib4_client import syntheticlib4_client +from backend.shared.syntheticlib4_client import SyntheticLib4Client, syntheticlib4_client class ProofSearchService: """Coordinates source normalization and the local SQLite proof-search index.""" - def __init__(self, index_path: Path | None = None) -> None: + def __init__( + self, + index_path: Path | None = None, + *, + syntheticlib4_source: SyntheticLib4Client | None = None, + ) -> None: self._explicit_index_path = Path(index_path) if index_path else None + self._syntheticlib4_source = syntheticlib4_source or syntheticlib4_client self._lock = asyncio.Lock() @property @@ -73,6 +80,7 @@ async def search_candidate_pool( pool_limit: int, exclude_corpus_scopes: list[str] | None = None, exclude_session_ids: list[str] | None = None, + exclude_run_ids: list[str] | None = None, ) -> list[UnifiedProofSearchRecord]: """Return a wider internal candidate pool for Assistant ranking. @@ -90,6 +98,34 @@ async def search_candidate_pool( pool_limit=pool_limit, exclude_corpus_scopes=exclude_corpus_scopes, exclude_session_ids=exclude_session_ids, + exclude_run_ids=exclude_run_ids, + ) + + async def exact_identity_neighborhood( + self, + *, + theorem_statement_hashes: list[str], + lean_code_hashes: list[str], + corpora: list[str], + exclude_run_ids: list[str] | None = None, + exclude_session_ids: list[str] | None = None, + identity_version: str = CANONICAL_PROOF_IDENTITY_VERSION, + limit: int = 256, + ) -> list[UnifiedProofSearchRecord]: + enabled = set(default_proof_search_corpora()) + filtered = [corpus for corpus in corpora if corpus in enabled] + if not filtered: + return [] + await self._ensure_index() + return await asyncio.to_thread( + ProofSearchIndexer(self.index_path).exact_identity_neighborhood, + theorem_statement_hashes=theorem_statement_hashes, + lean_code_hashes=lean_code_hashes, + corpora=filtered, + exclude_run_ids=exclude_run_ids, + exclude_session_ids=exclude_session_ids, + identity_version=identity_version, + limit=limit, ) async def get_record( @@ -98,6 +134,8 @@ async def get_record( corpus: str, proof_id: str, session_id: str | None = None, + search_id: str | None = None, + run_id: str | None = None, ) -> UnifiedProofSearchRecord | None: """Fetch one indexed proof and hydrate SyntheticLib4 fixture code when available.""" if corpus not in set(default_proof_search_corpora()): @@ -108,12 +146,15 @@ async def get_record( corpus=corpus, proof_id=proof_id, session_id=session_id, + search_id=search_id, + run_id=run_id, ) + if record is None or record.corpus != "syntheticlib4" or record.lean_code: return record hydrated = await asyncio.to_thread( - syntheticlib4_client.hydrate_proof, + self._syntheticlib4_source.hydrate_proof, record.external_fingerprint or record.proof_id, ) if not hydrated or not str(hydrated.get("lean_code") or "").strip(): @@ -130,9 +171,39 @@ async def get_record( raise ValueError("SyntheticLib4 hydration Lean-code hash mismatch") return hydrated_record + async def support_lineage( + self, + *, + theorem_statement_hash: str, + lean_code_hash: str, + corpora: list[str], + exclude_run_ids: list[str], + exclude_session_ids: list[str], + offset: int, + limit: int, + ) -> tuple[int, list[dict[str, str]]]: + enabled = set(default_proof_search_corpora()) + filtered = [corpus for corpus in corpora if corpus in enabled] + if not filtered: + return 0, [] + await self._ensure_index() + return await asyncio.to_thread( + ProofSearchIndexer(self.index_path).support_lineage, + theorem_statement_hash=theorem_statement_hash, + lean_code_hash=lean_code_hash, + corpora=filtered, + exclude_run_ids=exclude_run_ids, + exclude_session_ids=exclude_session_ids, + offset=offset, + limit=limit, + ) + async def _ensure_index(self) -> None: - if self.index_path.exists() and not await asyncio.to_thread( - self._sources_are_newer_than_index + indexer = ProofSearchIndexer(self.index_path) + if ( + self.index_path.exists() + and await asyncio.to_thread(indexer.is_compatible) + and not await asyncio.to_thread(self._sources_are_newer_than_index) ): return await self.rebuild_index() @@ -178,7 +249,9 @@ def _sources_are_newer_than_index(self) -> bool: async def _load_records(self) -> list[UnifiedProofSearchRecord]: records: list[UnifiedProofSearchRecord] = [] try: - records.extend(load_syntheticlib4_fixture_records()) + records.extend( + load_syntheticlib4_fixture_records(self._syntheticlib4_source) + ) except Exception: # SyntheticLib4 is optional; local MOTO proof search should still work. records.extend([]) diff --git a/backend/shared/proof_search/syntheticlib4_sources.py b/backend/shared/proof_search/syntheticlib4_sources.py index aa3a167..94d8d90 100644 --- a/backend/shared/proof_search/syntheticlib4_sources.py +++ b/backend/shared/proof_search/syntheticlib4_sources.py @@ -5,6 +5,7 @@ from typing import Any from backend.shared.proof_search.models import UnifiedProofSearchRecord +from backend.shared.proof_identity import canonical_proof_identity from backend.shared.syntheticlib4_client import SyntheticLib4Client, syntheticlib4_client @@ -26,6 +27,7 @@ def normalize_syntheticlib4_record( ) module = str(record.get("module", "") or "") source_path = str(record.get("source_path", "") or "") + internal_identity = canonical_proof_identity(theorem_statement, lean_code) return UnifiedProofSearchRecord( search_id=f"syntheticlib4:{fingerprint}", @@ -47,6 +49,9 @@ def normalize_syntheticlib4_record( lean_code=lean_code, lean_code_hash=lean_hash, theorem_statement_hash=statement_hash, + canonical_identity_version=internal_identity.version, + canonical_lean_code_hash=internal_identity.lean_code_hash if lean_code else "", + canonical_theorem_statement_hash=internal_identity.theorem_statement_hash, imports=_string_list(record.get("imports")), dependency_names=_string_list(record.get("dependency_names")), topic_tags=_string_list(record.get("topic_tags")), @@ -66,6 +71,9 @@ def normalize_syntheticlib4_record( "license_terms_id": record.get("license_terms_id", ""), "release_membership": record.get("release_membership", ""), "hydration_url": record.get("hydration_url"), + "canonical_identity_version": internal_identity.version, + "canonical_theorem_statement_hash": internal_identity.theorem_statement_hash, + "canonical_lean_code_hash": internal_identity.lean_code_hash if lean_code else "", }, ) diff --git a/backend/shared/provider_errors.py b/backend/shared/provider_errors.py new file mode 100644 index 0000000..569b120 --- /dev/null +++ b/backend/shared/provider_errors.py @@ -0,0 +1,136 @@ +"""Safe typed errors for model-provider routing failures.""" +from __future__ import annotations + +from dataclasses import dataclass, replace +from typing import Optional + +from backend.shared.log_redaction import redact_log_text + + +@dataclass(frozen=True) +class ProviderRouteIdentity: + """Non-secret identity for the route that produced a model error.""" + + provider: str + model: str + role_id: str = "" + task_id: str = "" + host_provider: str = "" + route_kind: str = "primary" + configured_provider: str = "" + configured_model: str = "" + + def with_context( + self, + *, + role_id: Optional[str] = None, + task_id: Optional[str] = None, + route_kind: Optional[str] = None, + configured_provider: Optional[str] = None, + configured_model: Optional[str] = None, + ) -> "ProviderRouteIdentity": + return replace( + self, + role_id=self.role_id if role_id is None else role_id, + task_id=self.task_id if task_id is None else task_id, + route_kind=self.route_kind if route_kind is None else route_kind, + configured_provider=( + self.configured_provider + if configured_provider is None + else configured_provider + ), + configured_model=( + self.configured_model + if configured_model is None + else configured_model + ), + ) + + def safe_summary(self) -> str: + parts = [ + f"provider={redact_log_text(self.provider or 'unknown', 80)}", + f"model={redact_log_text(self.model or 'unknown', 160)}", + f"route={redact_log_text(self.route_kind or 'primary', 40)}", + ] + if self.host_provider: + parts.append(f"host={redact_log_text(self.host_provider, 80)}") + if self.role_id: + parts.append(f"role={redact_log_text(self.role_id, 120)}") + if self.task_id: + parts.append(f"task={redact_log_text(self.task_id, 120)}") + return ", ".join(parts) + + +class ProviderRouteError(RuntimeError): + """A provider failure carrying safe route identity and its original cause.""" + + def __init__( + self, + message: str, + *, + route: ProviderRouteIdentity, + cause: Optional[BaseException] = None, + ) -> None: + self.route = route + self.cause = cause + self.safe_message = redact_log_text(message, 1000) + super().__init__(f"{self.safe_message} ({route.safe_summary()})") + + def with_route_context( + self, + *, + role_id: str, + task_id: str, + route_kind: Optional[str] = None, + configured_provider: Optional[str] = None, + configured_model: Optional[str] = None, + ) -> "ProviderRouteError": + return type(self)( + self.safe_message, + route=self.route.with_context( + role_id=role_id, + task_id=task_id, + route_kind=route_kind, + configured_provider=configured_provider, + configured_model=configured_model, + ), + cause=self.cause, + ) + + +class ProviderContextLengthError(ValueError): + """A provider rejected mandatory input because it exceeded context capacity.""" + + def __init__( + self, + message: str, + *, + route: ProviderRouteIdentity, + cause: Optional[BaseException] = None, + ) -> None: + self.route = route + self.cause = cause + self.safe_message = redact_log_text(message, 1000) + super().__init__(f"{self.safe_message} ({route.safe_summary()})") + + def with_route_context( + self, + *, + role_id: str, + task_id: str, + route_kind: Optional[str] = None, + configured_provider: Optional[str] = None, + configured_model: Optional[str] = None, + ) -> "ProviderContextLengthError": + return type(self)( + self.safe_message, + route=self.route.with_context( + role_id=role_id, + task_id=task_id, + route_kind=route_kind, + configured_provider=configured_provider, + configured_model=configured_model, + ), + cause=self.cause, + ) + diff --git a/backend/shared/rag_lock.py b/backend/shared/rag_lock.py index de6848f..0e5e1b4 100644 --- a/backend/shared/rag_lock.py +++ b/backend/shared/rag_lock.py @@ -3,6 +3,7 @@ """ import asyncio import logging +from contextlib import asynccontextmanager logger = logging.getLogger(__name__) @@ -46,9 +47,16 @@ def release(self): Release lock. Only fully releases when acquisition count reaches 0 (handles reentrant acquisitions). """ + current_task = asyncio.current_task() if self._acquisition_count <= 0: logger.warning("Attempted to release RAG lock when not held") - return + return False + if self._current_task is not current_task: + logger.error( + "Refusing RAG lock release by non-owner task (holder=%s)", + self._current_holder, + ) + return False self._acquisition_count -= 1 @@ -59,6 +67,22 @@ def release(self): self._lock.release() else: logger.debug(f"RAG lock reentrant release (count={self._acquisition_count})") + return True + + def owned_by_current_task(self) -> bool: + return ( + self._acquisition_count > 0 + and self._current_task is asyncio.current_task() + ) + + @asynccontextmanager + async def operation(self, operation_name: str): + """Acquire and owner-safely release one reentrant RAG operation.""" + await self.acquire(operation_name) + try: + yield self + finally: + self.release() async def __aenter__(self): await self.acquire("context_manager") diff --git a/backend/shared/runtime_root_lock.py b/backend/shared/runtime_root_lock.py new file mode 100644 index 0000000..ca17291 --- /dev/null +++ b/backend/shared/runtime_root_lock.py @@ -0,0 +1,122 @@ +"""Authoritative cross-process ownership for one mutable MOTO data root.""" +from __future__ import annotations + +from contextlib import contextmanager +import json +import os +from pathlib import Path +import threading +from typing import Iterator + + +class RuntimeRootInUseError(RuntimeError): + """Raised when another backend already owns the requested data root.""" + + +_PROCESS_GUARD = threading.Lock() +_PROCESS_OWNED: set[str] = set() + + +class RuntimeRootLease: + def __init__(self, data_root: str | Path) -> None: + self.data_root = Path(data_root).resolve() + self.lock_path = self.data_root / ".moto_backend.lock" + self._file = None + self._identity = os.path.normcase(os.path.realpath(str(self.data_root))) + + def acquire(self) -> "RuntimeRootLease": + self.data_root.mkdir(parents=True, exist_ok=True) + with _PROCESS_GUARD: + if self._identity in _PROCESS_OWNED: + raise RuntimeRootInUseError( + f"MOTO data root is already owned by this backend process: {self.data_root}" + ) + _PROCESS_OWNED.add(self._identity) + + file_obj = None + try: + file_obj = self.lock_path.open("a+b") + file_obj.seek(0) + if file_obj.read(1) == b"": + file_obj.seek(0) + file_obj.write(b"\0") + file_obj.flush() + file_obj.seek(0) + self._lock_file(file_obj) + payload = json.dumps( + {"pid": os.getpid(), "data_root": str(self.data_root)}, + separators=(",", ":"), + ).encode("utf-8") + file_obj.seek(1) + file_obj.write(payload) + file_obj.truncate() + file_obj.flush() + self._file = file_obj + return self + except BaseException as exc: + if file_obj is not None: + file_obj.close() + with _PROCESS_GUARD: + _PROCESS_OWNED.discard(self._identity) + if isinstance(exc, RuntimeRootInUseError): + raise + raise RuntimeRootInUseError( + f"MOTO data root is already in use by another backend process: {self.data_root}" + ) from exc + + @staticmethod + def _lock_file(file_obj) -> None: + if os.name == "nt": + import msvcrt + + try: + msvcrt.locking(file_obj.fileno(), msvcrt.LK_NBLCK, 1) + except OSError as exc: + raise RuntimeRootInUseError("Windows runtime-root lock is held") from exc + else: + import fcntl + + try: + fcntl.flock(file_obj.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB) + except OSError as exc: + raise RuntimeRootInUseError("POSIX runtime-root lock is held") from exc + + def release(self) -> None: + file_obj = self._file + self._file = None + try: + if file_obj is not None: + file_obj.seek(0) + if os.name == "nt": + import msvcrt + + try: + msvcrt.locking(file_obj.fileno(), msvcrt.LK_UNLCK, 1) + except OSError: + # Closing the descriptor also releases all Windows byte + # locks. This covers runtimes that reject explicit + # unlock after writes changed the current file pointer. + pass + else: + import fcntl + + fcntl.flock(file_obj.fileno(), fcntl.LOCK_UN) + file_obj.close() + finally: + with _PROCESS_GUARD: + _PROCESS_OWNED.discard(self._identity) + + def __enter__(self) -> "RuntimeRootLease": + return self.acquire() + + def __exit__(self, _exc_type, _exc, _tb) -> None: + self.release() + + +@contextmanager +def hold_runtime_root(data_root: str | Path) -> Iterator[RuntimeRootLease]: + lease = RuntimeRootLease(data_root).acquire() + try: + yield lease + finally: + lease.release() diff --git a/backend/shared/runtime_settings.py b/backend/shared/runtime_settings.py index bd2979e..bd91101 100644 --- a/backend/shared/runtime_settings.py +++ b/backend/shared/runtime_settings.py @@ -38,6 +38,7 @@ class RuntimeSettingsError(RuntimeError): } _LEGACY_DEFAULT_LEAN4_PROOF_TIMEOUT = 600 +_LEGACY_DEFAULT_SMT_TIMEOUT = 30 _CONNECTIVITY_BOOL_FIELDS = { "syntheticlib4_enabled", @@ -128,8 +129,8 @@ def _proof_settings_from_config() -> Dict[str, Any]: def _free_model_settings_from_manager() -> Dict[str, Any]: status = free_model_manager.get_status() return { - "looping_enabled": bool(status.get("looping_enabled", True)), - "auto_selector_enabled": bool(status.get("auto_selector_enabled", True)), + "looping_enabled": bool(status.get("looping_enabled", False)), + "auto_selector_enabled": bool(status.get("auto_selector_enabled", False)), } @@ -199,6 +200,12 @@ def apply_persisted_runtime_settings() -> None: and str(proof_settings[field]).strip() == str(_LEGACY_DEFAULT_LEAN4_PROOF_TIMEOUT) ): proof_settings[field] = default + if ( + field == "smt_timeout" + and default != _LEGACY_DEFAULT_SMT_TIMEOUT + and str(proof_settings[field]).strip() == str(_LEGACY_DEFAULT_SMT_TIMEOUT) + ): + proof_settings[field] = default setattr( system_config, field, diff --git a/backend/shared/sleep_inhibitor.py b/backend/shared/sleep_inhibitor.py new file mode 100644 index 0000000..2f9f003 --- /dev/null +++ b/backend/shared/sleep_inhibitor.py @@ -0,0 +1,164 @@ +"""Best-effort desktop sleep inhibition for active top-level workflows.""" +from __future__ import annotations + +import ctypes +import logging +import sys +import threading +import time +from typing import Callable, Hashable, Optional + +from backend.shared.config import system_config + +logger = logging.getLogger(__name__) + +ES_SYSTEM_REQUIRED = 0x00000001 +ES_CONTINUOUS = 0x80000000 + + +class SleepInhibitor: + """Keep Windows awake while logical workflow owners are active. + + The public methods only update in-memory desired state. One persistent + worker owns all Windows calls, preserving SetThreadExecutionState's + thread-affinity without blocking the FastAPI event loop. + """ + + def __init__( + self, + *, + platform: Optional[str] = None, + execution_state_setter: Optional[Callable[[int], int]] = None, + ) -> None: + self._platform = sys.platform if platform is None else platform + self._execution_state_setter = execution_state_setter + self._owners: set[Hashable] = set() + self._lock = threading.Lock() + self._state_changed = threading.Event() + self._worker: Optional[threading.Thread] = None + self._desired_active = False + self._native_active = False + self._worker_generation = 0 + + def _is_enabled(self) -> bool: + return self._platform == "win32" and not system_config.generic_mode + + def _set_execution_state(self, flags: int) -> int: + setter = self._execution_state_setter + if setter is None: + setter = ctypes.windll.kernel32.SetThreadExecutionState + return int(setter(flags)) + + def _run_worker(self, generation: int) -> None: + while True: + self._state_changed.wait() + self._state_changed.clear() + with self._lock: + if generation != self._worker_generation: + return + desired_active = self._desired_active + native_active = self._native_active + if desired_active == native_active: + continue + + flags = ( + ES_CONTINUOUS | ES_SYSTEM_REQUIRED + if desired_active + else ES_CONTINUOUS + ) + try: + succeeded = self._set_execution_state(flags) != 0 + except Exception: + succeeded = False + logger.exception("Unable to update Windows sleep inhibition") + + with self._lock: + if generation != self._worker_generation: + return + if succeeded: + self._native_active = desired_active + if succeeded: + logger.info( + "Windows automatic sleep inhibition %s", + "active" if desired_active else "released", + ) + continue + + logger.warning( + "Windows SetThreadExecutionState failed; %s will be retried", + "sleep inhibition" if desired_active else "sleep-state restoration", + ) + time.sleep(1) + self._state_changed.set() + + def _ensure_worker_locked(self) -> None: + if self._worker and self._worker.is_alive(): + return + self._worker_generation += 1 + generation = self._worker_generation + try: + worker = threading.Thread( + target=self._run_worker, + args=(generation,), + name="moto-sleep-inhibitor", + daemon=True, + ) + worker.start() + self._worker = worker + except Exception: + self._worker = None + logger.exception( + "Unable to start Windows sleep-inhibitor worker; workflow will continue" + ) + + def acquire(self, owner: Hashable) -> None: + """Register an owner and inhibit idle sleep when the first owner arrives.""" + if not owner: + raise ValueError("Sleep inhibitor owner must be non-empty") + if not self._is_enabled(): + return + with self._lock: + if owner in self._owners: + return + self._owners.add(owner) + self._desired_active = True + self._ensure_worker_locked() + self._state_changed.set() + + def release(self, owner: Hashable) -> None: + """Remove an owner and restore normal sleep behavior after the last owner.""" + if not self._is_enabled(): + return + with self._lock: + if owner not in self._owners: + return + self._owners.remove(owner) + if self._owners: + return + self._desired_active = False + self._ensure_worker_locked() + self._state_changed.set() + + def release_all(self) -> None: + """Clear every owner and restore normal sleep behavior.""" + if not self._is_enabled(): + return + with self._lock: + self._owners.clear() + self._desired_active = False + if self._native_active: + self._ensure_worker_locked() + self._state_changed.set() + + @property + def owners(self) -> frozenset[Hashable]: + with self._lock: + return frozenset(self._owners) + + @property + def native_active(self) -> bool: + with self._lock: + return self._native_active + + +sleep_inhibitor = SleepInhibitor() diff --git a/backend/shared/solution_path/__init__.py b/backend/shared/solution_path/__init__.py new file mode 100644 index 0000000..90579d2 --- /dev/null +++ b/backend/shared/solution_path/__init__.py @@ -0,0 +1,61 @@ +"""Shared durable solution-path planning primitives.""" + +from .engine import MIN_ACCEPTANCES, ProposalReviewer, SolutionPathEngine +from .models import ( + AuditRecord, + DurableSolutionPathState, + PlanProposal, + ProposalStatus, + RepairReason, + ReviewDecision, + RouteEdit, + RouteStep, + SolutionPlan, + SolutionRoute, + StepOrdering, +) +from .integration import ( + advisory_plan_block, + enqueue_optional_update, + note_acceptances, + validator_instruction, + with_budgeted_solver_plan, + with_solver_plan, + with_validator_hook, +) +from .registry import ( + SolutionPathManagerRegistry, + solution_path_registry, + stable_solution_path_run_id, +) +from .reviewer import build_review_prompt, compact_review_prompt, review_with_json_retry + +__all__ = [ + "MIN_ACCEPTANCES", + "AuditRecord", + "DurableSolutionPathState", + "PlanProposal", + "ProposalReviewer", + "ProposalStatus", + "RepairReason", + "ReviewDecision", + "RouteEdit", + "RouteStep", + "SolutionPathEngine", + "SolutionPlan", + "SolutionRoute", + "StepOrdering", + "SolutionPathManagerRegistry", + "advisory_plan_block", + "enqueue_optional_update", + "note_acceptances", + "validator_instruction", + "solution_path_registry", + "stable_solution_path_run_id", + "with_budgeted_solver_plan", + "with_solver_plan", + "with_validator_hook", + "build_review_prompt", + "compact_review_prompt", + "review_with_json_retry", +] diff --git a/backend/shared/solution_path/engine.py b/backend/shared/solution_path/engine.py new file mode 100644 index 0000000..d190b6b --- /dev/null +++ b/backend/shared/solution_path/engine.py @@ -0,0 +1,872 @@ +"""Run-scoped, crash-safe solution path proposal and review engine.""" + +from __future__ import annotations + +import asyncio +from datetime import timedelta +import json +import os +import random +from pathlib import Path +from typing import Awaitable, Callable, Protocol + +from backend.shared.api_client_manager import RetryableProviderError +from backend.shared.model_error_utils import is_non_retryable_model_error +from backend.shared.provider_pause import ( + is_provider_credit_pause_error, + mark_provider_paused, + wait_for_provider_resume, +) + +from .models import ( + AuditRecord, + DurableSolutionPathState, + PlanProposal, + ProposalStatus, + RepairReason, + ReviewDecision, + SolutionPlan, + SolutionRoute, + RouteStep, + utc_now, +) + +MIN_ACCEPTANCES = 5 +MAX_AUDIT_RECORDS = 500 +MAX_PROPOSALS = 100 +MAX_REVIEW_FAILURES = 12 +MAX_AUTOMATIC_REVIEW_FAILURES = 3 +RETRY_BASE_SECONDS = 1.0 +RETRY_MAX_SECONDS = 300.0 + + +class ProposalReviewer(Protocol): + async def __call__( + self, proposal: PlanProposal, current_plan: SolutionPlan | None + ) -> ReviewDecision: ... + + +Reviewer = Callable[ + [PlanProposal, SolutionPlan | None], Awaitable[ReviewDecision] +] + + +class SolutionPathEngine: + """Owns exactly one plan and a serial durable proposal queue for one run.""" + + def __init__( + self, + root: Path | str, + run_id: str, + reviewer: Reviewer, + *, + workflow_mode: str = "", + user_prompt: str = "", + retry_base_seconds: float = RETRY_BASE_SECONDS, + retry_max_seconds: float = RETRY_MAX_SECONDS, + ): + if not run_id or Path(run_id).name != run_id: + raise ValueError("run_id must be a non-empty single path component") + self.run_id = run_id + self._reviewer = reviewer + self._directory = Path(root) / run_id + self._state_path = self._directory / "solution_path_state.json" + self._generation_path = ( + Path(root) / ".solution_path_generations" / f"{run_id}.json" + ) + self._lock = asyncio.Lock() + self._worker: asyncio.Task[None] | None = None + self._stopped = False + self._reviewer_generation = 1 + self._retry_base_seconds = max(0.01, retry_base_seconds) + self._retry_max_seconds = max(self._retry_base_seconds, retry_max_seconds) + self._state = self._load() + self._last_persisted_state = self._state.model_copy(deep=True) + if workflow_mode: + if self._state.workflow_mode and self._state.workflow_mode != workflow_mode: + raise ValueError("persisted solution path workflow_mode mismatch") + self._state.workflow_mode = workflow_mode + if user_prompt: + from .models import prompt_fingerprint + canonical_prompt = user_prompt.strip() + fingerprint = prompt_fingerprint(canonical_prompt) + if self._state.prompt_hash and self._state.prompt_hash != fingerprint: + raise ValueError("persisted solution path user_prompt mismatch") + self._state.user_prompt = canonical_prompt + self._state.prompt_hash = fingerprint + + @property + def active(self) -> bool: + return self._state.acceptance_count >= MIN_ACCEPTANCES + + @property + def state(self) -> DurableSolutionPathState: + return self._state.model_copy(deep=True) + + async def set_reviewer(self, reviewer: Reviewer) -> None: + """Refresh runtime model configuration while retaining durable run state.""" + async with self._lock: + self._reviewer = reviewer + self._reviewer_generation += 1 + self._audit( + "reviewer_replaced", + detail=f"reviewer_generation={self._reviewer_generation}", + ) + await self._persist() + + async def set_acceptance_count(self, count: int) -> None: + if count < 0: + raise ValueError("acceptance count cannot be negative") + activation_payload = None + async with self._lock: + previous = self._state.acceptance_count + effective_count = max(previous, count) + if effective_count == previous: + return + self._state.acceptance_count = effective_count + self._audit("acceptance_count_updated", detail=str(effective_count)) + if previous < MIN_ACCEPTANCES <= effective_count: + activation_payload = self._event_payload( + message="Progressive solution-path tracking is now active." + ) + await self._persist() + if activation_payload is not None: + await self._broadcast( + "solution_path_activated", + activation_payload, + ) + self._ensure_worker() + + async def propose( + self, + route: SolutionRoute, + *, + lifecycle_generation: int | None = None, + rationale: str = "", + main_route: str = "", + proposer_role: str = "validator", + source_task_id: str | None = None, + source_phase: str | None = None, + source_decision: str | None = None, + ) -> PlanProposal: + async with self._lock: + if self._stopped: + raise ValueError("solution-path lifecycle is stopped") + if ( + lifecycle_generation is not None + and lifecycle_generation != self._state.lifecycle_generation + ): + raise ValueError("stale solution-path lifecycle generation") + base_revision = self._state.plan.revision if self._state.plan else 0 + proposal = PlanProposal( + run_id=self.run_id, + base_revision=base_revision, + main_route=main_route, + route=route, + rationale=rationale, + proposer_role=proposer_role, + source_task_id=source_task_id, + source_phase=source_phase, + source_decision=source_decision, + ) + self._state.proposals.append(proposal) + self._trim_proposals() + self._audit( + "proposal_queued", + proposal.proposal_id, + base_revision, + actor=proposer_role, + source_task_id=source_task_id, + ) + await self._persist() + event_payload = self._event_payload( + proposal=proposal, + message="A validator-proposed solution-path update was queued for Main Submitter 1 review.", + ) + await self._broadcast("solution_path_proposal_queued", event_payload) + self._ensure_worker() + return proposal.model_copy(deep=True) + + async def stop(self) -> None: + self._stopped = True + worker = self._worker + if worker and not worker.done(): + worker.cancel() + await asyncio.gather(worker, return_exceptions=True) + async with self._lock: + for proposal in self._state.proposals: + if proposal.status == ProposalStatus.REVIEWING: + proposal.status = ProposalStatus.QUEUED + proposal.updated_at = utc_now() + self._audit("engine_stopped") + await self._persist() + + async def start(self) -> None: + async with self._lock: + self._stopped = False + self._state.lifecycle_generation += 1 + for proposal in self._state.proposals: + if proposal.status == ProposalStatus.USER_REPAIR_REQUIRED: + proposal.repair_generation = self._state.lifecycle_generation + self._audit("engine_started") + await self._persist() + self._ensure_worker() + + async def clear(self) -> None: + await self.stop() + async with self._lock: + generation = self._state.lifecycle_generation + 1 + self._state = DurableSolutionPathState( + run_id=self.run_id, + workflow_mode=self._state.workflow_mode, + user_prompt=self._state.user_prompt, + prompt_hash=self._state.prompt_hash, + lifecycle_generation=generation, + ) + self._audit("engine_cleared") + await self._persist() + await asyncio.to_thread(self._write_generation_tombstone, generation) + await asyncio.to_thread(self._delete_state_files) + + async def resume_proposal( + self, + proposal_id: str, + *, + lifecycle_generation: int, + ) -> PlanProposal: + """Explicitly resume one repair-blocked proposal in the same generation.""" + async with self._lock: + if lifecycle_generation != self._state.lifecycle_generation: + raise ValueError("stale solution-path lifecycle generation") + proposal = next( + ( + item + for item in self._state.proposals + if item.proposal_id == proposal_id + ), + None, + ) + if proposal is None: + raise ValueError("solution-path proposal not found") + if proposal.status != ProposalStatus.USER_REPAIR_REQUIRED: + raise ValueError("solution-path proposal is not repair-blocked") + if ( + proposal.repair_generation is not None + and proposal.repair_generation != lifecycle_generation + ): + raise ValueError("stale solution-path proposal repair generation") + proposal.status = ProposalStatus.QUEUED + proposal.failure_count = 0 + proposal.last_error_type = None + proposal.repair_reason = None + proposal.repair_detail = "" + proposal.repair_generation = None + proposal.next_retry_at = None + proposal.feedback = "explicitly resumed after user repair" + proposal.updated_at = utc_now() + self._audit( + "proposal_resumed", + proposal.proposal_id, + detail=proposal.feedback, + ) + await self._persist() + payload = self._event_payload( + proposal=proposal, + message="The repaired solution-path proposal was explicitly resumed.", + ) + result = proposal.model_copy(deep=True) + await self._broadcast("solution_path_proposal_resumed", payload) + self._ensure_worker() + return result + + async def wait_idle(self) -> None: + while True: + worker = self._worker + if worker is None: + return + await asyncio.shield(worker) + if worker is self._worker: + return + + def _ensure_worker(self) -> None: + if self._stopped or not self.active: + return + if self._worker is None or self._worker.done(): + self._worker = asyncio.create_task(self._review_loop()) + + async def _review_loop(self) -> None: + while not self._stopped and self.active: + async with self._lock: + proposal = next( + ( + item + for item in self._state.proposals + if item.status in { + ProposalStatus.QUEUED, + ProposalStatus.FOLLOWUP, + ProposalStatus.REVIEWING, + } + ), + None, + ) + if proposal is None: + return + if proposal.next_retry_at is not None: + delay = (proposal.next_retry_at - utc_now()).total_seconds() + if delay > 0: + retry_generation = self._state.lifecycle_generation + else: + delay = 0 + else: + delay = 0 + if delay > 0: + proposal_snapshot = None + else: + retry_generation = self._state.lifecycle_generation + reviewer_generation = self._reviewer_generation + reviewer = self._reviewer + proposal.status = ProposalStatus.REVIEWING + proposal.review_count += 1 + proposal.updated_at = utc_now() + plan_snapshot = ( + self._state.plan.model_copy(deep=True) + if self._state.plan + else None + ) + proposal_snapshot = proposal.model_copy(deep=True) + self._audit( + "review_started", + proposal.proposal_id, + detail=f"reviewer_generation={reviewer_generation}", + ) + await self._persist() + if delay > 0: + await asyncio.sleep(min(delay, 1.0)) + continue + + await self._broadcast( + "solution_path_proposal_reviewing", + self._event_payload( + proposal=proposal_snapshot, + message="Main Submitter 1 is reviewing a proposed solution-path update.", + ), + ) + try: + decision = await reviewer(proposal_snapshot, plan_snapshot) + except asyncio.CancelledError: + raise + except Exception as exc: + if is_provider_credit_pause_error(exc): + mark_provider_paused() + async with self._lock: + proposal.status = ProposalStatus.QUEUED + proposal.last_error_type = "provider_credit_pause" + proposal.feedback = "review paused until provider credits are reset" + proposal.updated_at = utc_now() + self._audit( + "review_provider_paused", + proposal.proposal_id, + detail=proposal.feedback, + ) + await self._persist() + await wait_for_provider_resume(lambda: self._stopped) + continue + if self._is_context_overflow(exc): + async with self._lock: + if retry_generation != self._state.lifecycle_generation: + continue + self._mark_repair_required( + proposal, + RepairReason.CONTEXT_OVERFLOW, + exc, + ) + self._audit( + "review_user_repair_required", + proposal.proposal_id, + detail=proposal.feedback, + ) + await self._persist() + repair_payload = self._event_payload( + proposal=proposal, + message=( + "Solution-path review needs a larger context window or " + "a shorter user prompt before it can continue." + ), + detail=proposal.feedback, + ) + await self._broadcast( + "solution_path_proposal_user_repair_required", repair_payload + ) + continue + repair_reason = self._hard_failure_reason(exc) + if repair_reason is not None: + async with self._lock: + if retry_generation != self._state.lifecycle_generation: + continue + self._mark_repair_required(proposal, repair_reason, exc) + self._audit( + "review_user_repair_required", + proposal.proposal_id, + detail=proposal.feedback, + ) + await self._persist() + repair_payload = self._event_payload( + proposal=proposal, + message="Solution-path review needs user configuration repair before it can continue.", + detail=proposal.repair_detail, + ) + await self._broadcast( + "solution_path_proposal_user_repair_required", + repair_payload, + ) + continue + async with self._lock: + if retry_generation != self._state.lifecycle_generation: + continue + proposal.status = ProposalStatus.QUEUED + proposal.failure_count += 1 + proposal.last_error_type = type(exc).__name__ + proposal.feedback = self._safe_error(exc) + proposal.updated_at = utc_now() + if proposal.failure_count >= MAX_AUTOMATIC_REVIEW_FAILURES: + self._mark_repair_required( + proposal, + RepairReason.UNKNOWN_REVIEWER_FAILURE, + exc, + ) + self._audit( + "review_user_repair_required", + proposal.proposal_id, + detail=proposal.feedback, + ) + await self._persist() + repair_payload = self._event_payload( + proposal=proposal, + message="Solution-path review repeatedly failed and now requires explicit user repair.", + detail=proposal.repair_detail, + ) + retry_payload = None + else: + retry_delay = self._retry_delay(proposal.failure_count) + proposal.next_retry_at = utc_now() + timedelta( + seconds=retry_delay + ) + event = ( + "review_retry_scheduled" + if isinstance(exc, RetryableProviderError) + else "review_failed_retry_scheduled" + ) + self._audit(event, proposal.proposal_id, detail=proposal.feedback) + await self._persist() + retry_payload = self._event_payload( + proposal=proposal, + message="Solution-path review failed and the update remains queued for retry.", + detail=proposal.feedback, + ) + if retry_payload is None: + await self._broadcast( + "solution_path_proposal_user_repair_required", + repair_payload, + ) + else: + await self._broadcast("solution_path_proposal_retry_queued", retry_payload) + continue + + async with self._lock: + if ( + retry_generation != self._state.lifecycle_generation + or self._stopped + ): + continue + current_revision = self._state.plan.revision if self._state.plan else 0 + if proposal.base_revision != current_revision: + proposal.base_revision = current_revision + proposal.status = ProposalStatus.QUEUED + proposal.feedback = "stale base revision; queued for re-review" + proposal.updated_at = utc_now() + self._audit("proposal_stale_requeued", proposal.proposal_id, current_revision) + await self._persist() + retry_payload = self._event_payload( + proposal=proposal, + message="A stale solution-path update was queued for review again.", + detail=proposal.feedback, + ) + await self._broadcast("solution_path_proposal_retry_queued", retry_payload) + continue + try: + self._apply_decision(proposal, decision) + except (TypeError, ValueError) as exc: + proposal.failure_count += 1 + proposal.last_error_type = type(exc).__name__ + proposal.feedback = self._safe_error(exc) + proposal.updated_at = utc_now() + if proposal.failure_count >= MAX_AUTOMATIC_REVIEW_FAILURES: + self._mark_repair_required( + proposal, + RepairReason.INVALID_REVIEW, + exc, + ) + self._audit( + "review_user_repair_required", + proposal.proposal_id, + detail=proposal.feedback, + ) + else: + proposal.status = ProposalStatus.QUEUED + proposal.next_retry_at = utc_now() + timedelta( + seconds=self._retry_delay(proposal.failure_count) + ) + self._audit( + "review_edit_validation_retry_scheduled", + proposal.proposal_id, + detail=proposal.feedback, + ) + await self._persist() + event = ( + "solution_path_updated" + if proposal.status == ProposalStatus.APPROVED + else "solution_path_proposal_rejected" + if proposal.status == ProposalStatus.REJECTED + else "solution_path_proposal_retry_queued" + ) + event_payload = self._event_payload( + proposal=proposal, + decision=decision.decision, + detail=decision.reasoning, + message=( + "Main Submitter 1 approved an update to the current solution path." + if proposal.status == ProposalStatus.APPROVED + else "Main Submitter 1 rejected a proposed solution-path update." + if proposal.status == ProposalStatus.REJECTED + else "Main Submitter 1 requested another solution-path revision; it remains queued." + ), + ) + await self._broadcast(event, event_payload) + + def _apply_decision(self, proposal: PlanProposal, decision: ReviewDecision) -> None: + working = proposal.model_copy(deep=True) + working.feedback = decision.reasoning + working.last_error_type = None + working.repair_reason = None + working.repair_detail = "" + working.repair_generation = None + working.next_retry_at = None + working.updated_at = utc_now() + if decision.decision == "reject": + working.status = ProposalStatus.REJECTED + self._commit_proposal(proposal, working) + self._audit("proposal_rejected", proposal.proposal_id, detail=decision.reasoning) + return + if decision.decision == "followup": + if decision.followup_route is not None: + working.route = decision.followup_route.model_copy(deep=True) + if decision.main_route is not None: + working.main_route = decision.main_route + self._apply_route_edits(working.route, decision.edits) + working = PlanProposal.model_validate(working.model_dump()) + working.status = ProposalStatus.FOLLOWUP + self._commit_proposal(proposal, working) + self._audit("proposal_followup", proposal.proposal_id, detail=decision.reasoning) + return + if decision.followup_route is not None: + working.route = decision.followup_route.model_copy(deep=True) + if decision.main_route is not None: + working.main_route = decision.main_route + self._apply_route_edits(working.route, decision.edits) + working = PlanProposal.model_validate(working.model_dump()) + if decision.more_edits: + working.status = ProposalStatus.FOLLOWUP + self._commit_proposal(proposal, working) + self._audit("proposal_more_edits_requested", proposal.proposal_id) + return + revision = (self._state.plan.revision + 1) if self._state.plan else 1 + created_at = self._state.plan.created_at if self._state.plan else utc_now() + self._state.plan = SolutionPlan( + run_id=self.run_id, + revision=revision, + main_route=working.main_route, + route=working.route, + created_at=created_at, + ) + working.status = ProposalStatus.APPROVED + self._commit_proposal(proposal, working) + self._audit("proposal_approved", proposal.proposal_id, revision, decision.reasoning) + + @staticmethod + def _commit_proposal(target: PlanProposal, source: PlanProposal) -> None: + for field_name in type(target).model_fields: + setattr(target, field_name, getattr(source, field_name)) + + def _apply_route_edits(self, route: SolutionRoute, edits: list) -> None: + for edit in edits: + index = next( + (i for i, step in enumerate(route.steps) if step.step_id == edit.step_id), + None, + ) + if edit.operation == "check": + if index is None: + raise ValueError(f"checked route step not found: {edit.step_id}") + if ( + edit.expected_title is not None + and route.steps[index].title != edit.expected_title + ): + raise ValueError(f"checked route step title changed: {edit.step_id}") + elif edit.operation == "delete": + if index is None: + raise ValueError(f"route step to delete not found: {edit.step_id}") + route.steps.pop(index) + elif edit.operation == "update": + if index is None or edit.step is None: + raise ValueError(f"route step to update not found: {edit.step_id}") + replacement = edit.step.model_copy(deep=True) + replacement.step_id = edit.step_id + route.steps[index] = replacement + elif edit.operation == "add" and edit.step is not None: + new_step: RouteStep = edit.step.model_copy(deep=True) + if any(item.step_id == new_step.step_id for item in route.steps): + raise ValueError(f"duplicate route step id: {new_step.step_id}") + if edit.after_step_id is None: + route.steps.append(new_step) + else: + after = next( + ( + i + for i, item in enumerate(route.steps) + if item.step_id == edit.after_step_id + ), + None, + ) + if after is None: + raise ValueError( + f"route insertion anchor not found: {edit.after_step_id}" + ) + route.steps.insert(after + 1, new_step) + + def _event_payload( + self, + *, + message: str, + proposal: PlanProposal | None = None, + decision: str = "", + detail: str = "", + ) -> dict: + revision = self._state.plan.revision if self._state.plan else 0 + queued_statuses = {ProposalStatus.QUEUED, ProposalStatus.FOLLOWUP} + return { + "workflow_mode": self._state.workflow_mode or "unknown", + "run_id": self.run_id, + "acceptance_count": self._state.acceptance_count, + "prompt_hash": self._state.prompt_hash or None, + "lifecycle_generation": self._state.lifecycle_generation, + "revision": revision, + "proposal_id": proposal.proposal_id if proposal else None, + "proposal_status": proposal.status.value if proposal else None, + "base_revision": proposal.base_revision if proposal else revision, + "decision": decision or None, + "source_task_id": proposal.source_task_id if proposal else None, + "source_phase": proposal.source_phase if proposal else None, + "source_decision": proposal.source_decision if proposal else None, + "queued_proposals": sum( + item.status in queued_statuses for item in self._state.proposals + ), + "reviewing_proposals": sum( + item.status == ProposalStatus.REVIEWING + for item in self._state.proposals + ), + "repair_required_proposals": sum( + item.status == ProposalStatus.USER_REPAIR_REQUIRED + for item in self._state.proposals + ), + "repair_reason": ( + proposal.repair_reason.value + if proposal and proposal.repair_reason + else None + ), + "message": message, + "detail": detail, + } + + def _audit( + self, + event: str, + proposal_id: str | None = None, + plan_revision: int | None = None, + detail: str = "", + actor: str = "solution_path_engine", + source_task_id: str | None = None, + ) -> None: + self._state.audit.append( + AuditRecord( + sequence=self._state.next_audit_sequence, + event=event, + proposal_id=proposal_id, + plan_revision=plan_revision, + detail=detail, + actor=actor, + source_task_id=source_task_id, + ) + ) + self._state.next_audit_sequence += 1 + if len(self._state.audit) > MAX_AUDIT_RECORDS: + del self._state.audit[:-MAX_AUDIT_RECORDS] + + def _load(self) -> DurableSolutionPathState: + if not self._state_path.exists(): + generation = 1 + try: + payload = json.loads( + self._generation_path.read_text(encoding="utf-8") + ) + generation = max(generation, int(payload.get("lifecycle_generation", 1))) + except (FileNotFoundError, OSError, ValueError, TypeError): + pass + return DurableSolutionPathState( + run_id=self.run_id, + lifecycle_generation=generation, + ) + state = DurableSolutionPathState.model_validate_json( + self._state_path.read_text(encoding="utf-8") + ) + if state.run_id != self.run_id: + raise ValueError("persisted solution path run_id mismatch") + for proposal in state.proposals: + if proposal.status == ProposalStatus.REVIEWING: + proposal.status = ProposalStatus.QUEUED + if state.audit: + state.next_audit_sequence = max( + state.next_audit_sequence, + max(record.sequence for record in state.audit) + 1, + ) + return state + + async def _persist(self) -> None: + payload = self._state.model_dump(mode="json") + try: + await asyncio.to_thread(self._write_payload, payload) + except BaseException: + # Every caller mutates under _lock. Restoring the last successfully + # published snapshot makes those mutations transactional on I/O + # failure without exposing a half-persisted in-memory state. + self._state = self._last_persisted_state.model_copy(deep=True) + raise + self._last_persisted_state = self._state.model_copy(deep=True) + + def _write_payload(self, payload: dict) -> None: + self._directory.mkdir(parents=True, exist_ok=True) + temporary = self._state_path.with_suffix(".json.tmp") + with temporary.open("w", encoding="utf-8", newline="\n") as handle: + json.dump(payload, handle, ensure_ascii=False, indent=2) + handle.flush() + os.fsync(handle.fileno()) + os.replace(temporary, self._state_path) + + def _write_generation_tombstone(self, generation: int) -> None: + self._generation_path.parent.mkdir(parents=True, exist_ok=True) + temporary = self._generation_path.with_suffix(".json.tmp") + with temporary.open("w", encoding="utf-8", newline="\n") as handle: + json.dump( + { + "run_id": self.run_id, + "lifecycle_generation": generation, + }, + handle, + ensure_ascii=False, + indent=2, + ) + handle.flush() + os.fsync(handle.fileno()) + os.replace(temporary, self._generation_path) + + def _delete_state_files(self) -> None: + try: + self._state_path.unlink() + except FileNotFoundError: + pass + try: + self._directory.rmdir() + except (FileNotFoundError, OSError): + pass + + def _trim_proposals(self) -> None: + if len(self._state.proposals) <= MAX_PROPOSALS: + return + active = [ + proposal + for proposal in self._state.proposals + if proposal.status + in {ProposalStatus.QUEUED, ProposalStatus.REVIEWING, ProposalStatus.FOLLOWUP} + ] + terminal = [ + proposal + for proposal in self._state.proposals + if proposal.status + in { + ProposalStatus.APPROVED, + ProposalStatus.REJECTED, + ProposalStatus.USER_REPAIR_REQUIRED, + } + ] + keep_terminal = max(0, MAX_PROPOSALS - len(active)) + retained_terminal = terminal[-keep_terminal:] if keep_terminal else [] + self._state.proposals = retained_terminal + active + + def _retry_delay(self, failure_count: int) -> float: + exponent = min(max(0, failure_count - 1), MAX_REVIEW_FAILURES) + base = min(self._retry_max_seconds, self._retry_base_seconds * (2**exponent)) + return min(self._retry_max_seconds, base * random.uniform(0.8, 1.2)) + + def _mark_repair_required( + self, + proposal: PlanProposal, + reason: RepairReason, + exc: Exception, + ) -> None: + detail = self._safe_error(exc) + proposal.status = ProposalStatus.USER_REPAIR_REQUIRED + proposal.last_error_type = type(exc).__name__ + proposal.repair_reason = reason + proposal.repair_detail = detail + proposal.repair_generation = self._state.lifecycle_generation + proposal.feedback = detail + proposal.next_retry_at = None + proposal.updated_at = utc_now() + + @staticmethod + def _hard_failure_reason(exc: Exception) -> RepairReason | None: + if not is_non_retryable_model_error(exc): + return None + text = str(exc).lower() + if "api key" in text or "missing key" in text: + return RepairReason.MISSING_API_KEY + if "privacy" in text: + return RepairReason.PRIVACY_POLICY + return RepairReason.PROVIDER_CONFIGURATION + + @staticmethod + def _safe_error(exc: Exception) -> str: + text = " ".join(str(exc).split()) + return f"reviewer error: {type(exc).__name__}: {text[:500]}" + + @staticmethod + def _is_context_overflow(exc: Exception) -> bool: + try: + from backend.shared.provider_errors import ProviderContextLengthError + + if isinstance(exc, ProviderContextLengthError): + return True + except ImportError: + pass + text = str(exc).lower() + return ( + "solution-path reviewer mandatory context exceeds" in text + or "context_length_exceeded" in text + or "context window" in text and "exceed" in text + ) + + async def _broadcast(self, event: str, payload: dict) -> None: + try: + from backend.api.routes.websocket import broadcast_event + await broadcast_event(event, payload) + except Exception: + # Plan review remains secondary and must never block the parent workflow. + pass diff --git a/backend/shared/solution_path/integration.py b/backend/shared/solution_path/integration.py new file mode 100644 index 0000000..eb7393e --- /dev/null +++ b/backend/shared/solution_path/integration.py @@ -0,0 +1,212 @@ +"""Fault-tolerant hooks for attaching solution-path state to existing LLM calls.""" + +from __future__ import annotations + +import logging +import asyncio +from typing import Any, Protocol + +from pydantic import ValidationError + +from backend.shared.utils import count_tokens + +from .models import SolutionPlan, SolutionRoute + +logger = logging.getLogger(__name__) +PLAN_UPDATE_KEYS = ("solution_path_update", "progressive_solution_path") +_SUPERVISED_UPDATE_TASKS: set[asyncio.Task[Any]] = set() + + +def _supervise_optional_update(task: asyncio.Task[Any]) -> None: + """Retain background persistence and observe every terminal exception.""" + _SUPERVISED_UPDATE_TASKS.add(task) + + def done(completed: asyncio.Task[Any]) -> None: + _SUPERVISED_UPDATE_TASKS.discard(completed) + if completed.cancelled(): + logger.warning("Optional solution-path update persistence was cancelled") + return + try: + completed.result() + except Exception: + logger.exception("Unable to persist optional solution-path update") + + task.add_done_callback(done) + + +class SolutionPathManager(Protocol): + @property + def active(self) -> bool: ... + + @property + def state(self) -> Any: ... + + async def propose( + self, + route: SolutionRoute, + *, + lifecycle_generation: int, + rationale: str = "", + main_route: str = "", + proposer_role: str = "validator", + source_task_id: str | None = None, + source_phase: str | None = None, + source_decision: str | None = None, + ) -> Any: ... + + async def set_acceptance_count(self, count: int) -> None: ... + + +def validator_instruction(*, batch: bool = False) -> str: + placement = ( + "For a batch response, include at most one `solution_path_update` at the " + "top level beside `decisions`, never inside an individual decision. " + if batch + else "Include it only as a top-level field beside the primary decision fields. " + ) + return ( + "\n\nPROGRESSIVE SOLUTION PATH (OPTIONAL): Alongside your normal decision, " + "you MAY include `solution_path_update` when this same semantic review " + "reveals a material development: a stronger route or goal, evidence that " + "requires a substantial revision, or completion of a meaningful plan item. " + "If no approved solution path is shown above and the available context now " + "supports a clear, useful solution route, use `solution_path_update` to " + "propose the initial path; the absence of an existing path is not a reason " + "to omit a clear initial proposal. If an approved path is shown, continue " + "to use `solution_path_update` for material revisions or meaningful progress. " + "Do not update it for small nuances, wording changes, or routine validation. " + + placement + + "This is the sole permitted extension to the otherwise exact primary JSON " + "schema. Its compact shape is " + '`"solution_path_update":{"main_route":"non-empty distilled overall route",' + '"route":{"ordering":"ordered|parallel|mixed","steps":[...]},' + '"rationale":"optional"}`. `main_route` must be non-empty for an initial path; ' + "for a later update it may be omitted to preserve the approved main route. " + "Omit the entire extension otherwise. This field is advisory and must " + "not alter your primary decision. Do not use it for deterministic, tool, " + "syntax, proof-checker, or integrity checks." + ) + + +def advisory_plan_block(manager: SolutionPathManager | None) -> str: + if manager is None or not manager.active: + return "" + plan: SolutionPlan | None = getattr(manager.state, "plan", None) + if plan is None: + return "" + return ( + "\n\n[ADVISORY PROGRESSIVE SOLUTION PATH]\n" + "This is a distillation attempt at the best currently known path from " + "the available context. It may be wrong or incomplete, is guidance only, " + "and may be deviated from whenever a better route serves the user prompt " + "or the current subgoal. It is not authoritative evidence.\n" + f"{plan.model_dump_json(indent=2)}\n" + "[END ADVISORY PROGRESSIVE SOLUTION PATH]" + ) + + +def with_validator_hook( + prompt: str, + manager: SolutionPathManager | None, + *, + batch: bool = False, +) -> str: + if manager is None or not manager.active: + return prompt + return prompt + advisory_plan_block(manager) + validator_instruction(batch=batch) + + +def with_solver_plan(prompt: str, manager: SolutionPathManager | None) -> str: + return prompt + advisory_plan_block(manager) + + +def with_budgeted_solver_plan( + prompt: str, + manager: SolutionPathManager | None, + max_input_tokens: int, +) -> str: + """Append the advisory plan only when the complete prompt still fits.""" + block = advisory_plan_block(manager) + if not block: + return prompt + candidate = prompt + block + return candidate if count_tokens(candidate) <= max_input_tokens else prompt + + +async def enqueue_optional_update( + payload: Any, + manager: SolutionPathManager | None, + *, + proposer_role: str = "validator", + source_task_id: str | None = None, + source_phase: str | None = None, + source_decision: str | None = None, +) -> Any | None: + """Validate then supervise durable enqueue without blocking primary validation.""" + if manager is None or not manager.active or not isinstance(payload, dict): + return None + raw = next((payload.get(key) for key in PLAN_UPDATE_KEYS if key in payload), None) + if not isinstance(raw, dict): + return None + try: + route = SolutionRoute.model_validate(raw.get("route", raw)) + except (ValidationError, TypeError, ValueError): + logger.debug("Ignoring malformed optional solution-path update") + return None + rationale = raw.get("rationale", "") + if not isinstance(rationale, str): + rationale = "" + current_plan: SolutionPlan | None = getattr(manager.state, "plan", None) + if "main_route" in raw: + main_route = raw.get("main_route") + if not isinstance(main_route, str) or not main_route.strip(): + logger.debug("Ignoring optional solution-path update with invalid main_route") + return None + main_route = main_route.strip() + elif current_plan is not None and current_plan.main_route.strip(): + main_route = current_plan.main_route + else: + logger.debug("Ignoring initial solution-path update without main_route") + return None + lifecycle_generation = getattr(manager.state, "lifecycle_generation", None) + async def persist() -> Any | None: + try: + propose_kwargs = { + "rationale": rationale, + "main_route": main_route, + "proposer_role": proposer_role, + "source_task_id": source_task_id, + "source_phase": source_phase, + "source_decision": source_decision, + } + if lifecycle_generation is not None: + propose_kwargs["lifecycle_generation"] = int(lifecycle_generation) + return await manager.propose( + route, + **propose_kwargs, + ) + except Exception: + logger.exception("Unable to persist optional solution-path update") + return None + + task = asyncio.create_task( + persist(), + name=f"solution-path-update:{source_task_id or proposer_role}", + ) + _supervise_optional_update(task) + # Give cheap/in-memory managers one turn to finish while never waiting on disk I/O. + await asyncio.sleep(0) + if task.done() and not task.cancelled(): + try: + return task.result() + except Exception: + return None + return task + + +async def note_acceptances( + manager: SolutionPathManager | None, count: int +) -> None: + if manager is None: + return + await manager.set_acceptance_count(max(0, count)) diff --git a/backend/shared/solution_path/models.py b/backend/shared/solution_path/models.py new file mode 100644 index 0000000..670a90d --- /dev/null +++ b/backend/shared/solution_path/models.py @@ -0,0 +1,199 @@ +"""Typed durable records for the shared solution-path engine.""" + +from __future__ import annotations + +from datetime import datetime, timezone +from enum import Enum +import hashlib +from typing import Any, Literal +from uuid import uuid4 + +from pydantic import BaseModel, Field, field_validator, model_validator + + +MAX_ROUTE_STEPS = 32 +MAX_MAIN_ROUTE_CHARS = 8_000 +MAX_STEP_TITLE_CHARS = 500 +MAX_STEP_DESCRIPTION_CHARS = 8_000 +MAX_RATIONALE_CHARS = 8_000 +MAX_FEEDBACK_CHARS = 8_000 +MAX_METADATA_JSON_CHARS = 8_000 + + +def utc_now() -> datetime: + return datetime.now(timezone.utc) + + +def prompt_fingerprint(prompt: str) -> str: + return hashlib.sha256(prompt.strip().encode("utf-8")).hexdigest() + + +class StepOrdering(str, Enum): + ORDERED = "ordered" + UNORDERED = "unordered" + + +class RouteStep(BaseModel): + step_id: str = Field(default_factory=lambda: uuid4().hex, min_length=1, max_length=128) + title: str = Field(min_length=1, max_length=MAX_STEP_TITLE_CHARS) + description: str = Field(default="", max_length=MAX_STEP_DESCRIPTION_CHARS) + status: Literal["pending", "active", "complete", "blocked"] = "pending" + metadata: dict[str, Any] = Field(default_factory=dict) + + @field_validator("metadata") + @classmethod + def metadata_is_bounded_json(cls, value: dict[str, Any]) -> dict[str, Any]: + import json + + try: + encoded = json.dumps(value, ensure_ascii=False, sort_keys=True) + except (TypeError, ValueError) as exc: + raise ValueError("route step metadata must be JSON serializable") from exc + if len(encoded) > MAX_METADATA_JSON_CHARS: + raise ValueError( + f"route step metadata exceeds {MAX_METADATA_JSON_CHARS} characters" + ) + return value + + +class SolutionRoute(BaseModel): + ordering: StepOrdering = StepOrdering.ORDERED + steps: list[RouteStep] = Field(default_factory=list, max_length=MAX_ROUTE_STEPS) + + @field_validator("ordering", mode="before") + @classmethod + def normalize_parallel_ordering(cls, value: Any) -> Any: + if isinstance(value, str) and value.lower() in {"parallel", "mixed"}: + return StepOrdering.UNORDERED + return value + + @model_validator(mode="after") + def unique_step_ids(self) -> "SolutionRoute": + ids = [step.step_id for step in self.steps] + if len(ids) != len(set(ids)): + raise ValueError("route step_id values must be unique") + return self + + +class SolutionPlan(BaseModel): + run_id: str + revision: int = Field(default=1, ge=1) + main_route: str = Field(default="", max_length=MAX_MAIN_ROUTE_CHARS) + route: SolutionRoute + created_at: datetime = Field(default_factory=utc_now) + updated_at: datetime = Field(default_factory=utc_now) + + +class ProposalStatus(str, Enum): + QUEUED = "queued" + REVIEWING = "reviewing" + FOLLOWUP = "followup" + APPROVED = "approved" + REJECTED = "rejected" + USER_REPAIR_REQUIRED = "user_repair_required" + + +class RepairReason(str, Enum): + CONTEXT_OVERFLOW = "context_overflow" + MISSING_API_KEY = "missing_api_key" + PROVIDER_CONFIGURATION = "provider_configuration" + PRIVACY_POLICY = "privacy_policy" + INVALID_REVIEW = "invalid_review" + UNKNOWN_REVIEWER_FAILURE = "unknown_reviewer_failure" + + +class PlanProposal(BaseModel): + proposal_id: str = Field(default_factory=lambda: uuid4().hex) + run_id: str + base_revision: int = Field(ge=0) + main_route: str = Field(default="", max_length=MAX_MAIN_ROUTE_CHARS) + route: SolutionRoute + rationale: str = Field(default="", max_length=MAX_RATIONALE_CHARS) + status: ProposalStatus = ProposalStatus.QUEUED + review_count: int = Field(default=0, ge=0) + failure_count: int = Field(default=0, ge=0) + feedback: str = Field(default="", max_length=MAX_FEEDBACK_CHARS) + proposer_role: str = Field(default="validator", max_length=128) + source_task_id: str | None = Field(default=None, max_length=256) + source_phase: str | None = Field(default=None, max_length=128) + source_decision: Literal["accept", "reject", "mixed"] | None = None + last_error_type: str | None = None + repair_reason: RepairReason | None = None + repair_detail: str = Field(default="", max_length=MAX_FEEDBACK_CHARS) + repair_generation: int | None = Field(default=None, ge=1) + next_retry_at: datetime | None = None + created_at: datetime = Field(default_factory=utc_now) + updated_at: datetime = Field(default_factory=utc_now) + + +class RouteEdit(BaseModel): + """One checked edit to a proposal's working route.""" + + operation: Literal["add", "update", "delete", "check"] + step_id: str | None = None + step: RouteStep | None = None + after_step_id: str | None = None + expected_title: str | None = None + + @model_validator(mode="after") + def validate_edit_shape(self) -> "RouteEdit": + if self.operation == "add" and self.step is None: + raise ValueError("add edits require step") + if self.operation in {"update", "delete", "check"} and not self.step_id: + raise ValueError(f"{self.operation} edits require step_id") + if self.operation == "update" and self.step is None: + raise ValueError("update edits require step") + return self + + +class ReviewDecision(BaseModel): + decision: Literal["approve", "reject", "followup"] + reasoning: str = Field(default="", max_length=MAX_FEEDBACK_CHARS) + followup_route: SolutionRoute | None = None + main_route: str | None = None + edits: list[RouteEdit] = Field(default_factory=list, max_length=MAX_ROUTE_STEPS) + more_edits: bool = False + + @model_validator(mode="after") + def followup_has_route(self) -> "ReviewDecision": + if ( + self.decision == "followup" + and self.followup_route is None + and not self.edits + and self.main_route is None + ): + raise ValueError("followup decisions require a route, main_route, or edits") + if self.more_edits and self.decision == "reject": + raise ValueError("rejected decisions cannot request more edits") + return self + + +class AuditRecord(BaseModel): + sequence: int = Field(ge=1) + event: str + proposal_id: str | None = None + plan_revision: int | None = None + detail: str = "" + actor: str = "solution_path_engine" + source_task_id: str | None = None + timestamp: datetime = Field(default_factory=utc_now) + + +class DurableSolutionPathState(BaseModel): + schema_version: int = 2 + run_id: str + workflow_mode: str = "" + user_prompt: str = "" + prompt_hash: str = "" + lifecycle_generation: int = Field(default=1, ge=1) + acceptance_count: int = Field(default=0, ge=0) + plan: SolutionPlan | None = None + proposals: list[PlanProposal] = Field(default_factory=list) + audit: list[AuditRecord] = Field(default_factory=list) + next_audit_sequence: int = Field(default=1, ge=1) + + @model_validator(mode="after") + def normalize_prompt_provenance(self) -> "DurableSolutionPathState": + if self.user_prompt and not self.prompt_hash: + self.prompt_hash = prompt_fingerprint(self.user_prompt) + return self diff --git a/backend/shared/solution_path/registry.py b/backend/shared/solution_path/registry.py new file mode 100644 index 0000000..82ef282 --- /dev/null +++ b/backend/shared/solution_path/registry.py @@ -0,0 +1,247 @@ +"""Process-wide lifecycle registry for run-scoped solution-path managers.""" + +from __future__ import annotations + +import asyncio +import json +import os +import re +import shutil +from dataclasses import dataclass, field +from pathlib import Path + +from .engine import Reviewer, SolutionPathEngine +from .models import prompt_fingerprint + + +def stable_solution_path_run_id( + workflow_mode: str, + user_prompt: str, + *, + stable_run_id: str | None = None, +) -> str: + """Build a filesystem-safe identity without using mutable phase/source text.""" + if stable_run_id: + if Path(stable_run_id).name != stable_run_id: + raise ValueError("stable_run_id must be a single path component") + return stable_run_id + mode = re.sub(r"[^a-z0-9_-]+", "-", workflow_mode.lower()).strip("-") or "workflow" + return f"{mode}_{prompt_fingerprint(user_prompt)[:16]}" + + +@dataclass +class _RunLifecycle: + """Serialization and provenance shared by every operation on one run.""" + + root: Path + run_id: str + lock: asyncio.Lock = field(default_factory=asyncio.Lock) + generation: int = 0 + manager: SolutionPathEngine | None = None + workflow_mode: str | None = None + prompt_hash: str | None = None + + +class SolutionPathManagerRegistry: + def __init__(self) -> None: + self._runs: dict[tuple[str, str], _RunLifecycle] = {} + self._lock = asyncio.Lock() + self._last_acquired_identity: tuple[str, str] | None = None + + @staticmethod + def _canonical_root(root: Path | str) -> Path: + return Path(root).expanduser().resolve(strict=False) + + @classmethod + def _identity(cls, root: Path | str, run_id: str) -> tuple[str, str]: + if not run_id or Path(run_id).name != run_id: + raise ValueError("run_id must be a non-empty single path component") + canonical_root = cls._canonical_root(root) + return os.path.normcase(str(canonical_root)), run_id + + async def _lifecycle(self, root: Path | str, run_id: str) -> _RunLifecycle: + identity = self._identity(root, run_id) + async with self._lock: + lifecycle = self._runs.get(identity) + if lifecycle is None: + lifecycle = _RunLifecycle( + root=self._canonical_root(root), + run_id=run_id, + ) + self._runs[identity] = lifecycle + return lifecycle + + async def _matching_lifecycles(self, run_id: str) -> list[_RunLifecycle]: + async with self._lock: + return [ + lifecycle + for (_, candidate_run_id), lifecycle in self._runs.items() + if candidate_run_id == run_id + ] + + @staticmethod + def _validate_provenance( + lifecycle: _RunLifecycle, + *, + workflow_mode: str, + user_prompt: str, + ) -> None: + prompt_hash = prompt_fingerprint(user_prompt) + if ( + lifecycle.workflow_mode is not None + and lifecycle.workflow_mode != workflow_mode + ): + raise ValueError("loaded solution path workflow_mode mismatch") + if lifecycle.prompt_hash is not None and lifecycle.prompt_hash != prompt_hash: + raise ValueError("loaded solution path user_prompt mismatch") + + async def acquire( + self, + root: Path | str, + *, + workflow_mode: str, + user_prompt: str, + reviewer: Reviewer, + stable_run_id: str | None = None, + ) -> SolutionPathEngine: + user_prompt = user_prompt.strip() + run_id = stable_solution_path_run_id( + workflow_mode, user_prompt, stable_run_id=stable_run_id + ) + lifecycle = await self._lifecycle(root, run_id) + async with lifecycle.lock: + self._validate_provenance( + lifecycle, + workflow_mode=workflow_mode, + user_prompt=user_prompt, + ) + manager = lifecycle.manager + if manager is None: + manager = SolutionPathEngine( + lifecycle.root, + run_id, + reviewer, + workflow_mode=workflow_mode, + user_prompt=user_prompt, + ) + lifecycle.manager = manager + lifecycle.workflow_mode = workflow_mode + lifecycle.prompt_hash = prompt_fingerprint(user_prompt) + else: + await manager.set_reviewer(reviewer) + lifecycle.generation += 1 + await manager.start() + self._last_acquired_identity = self._identity(root, run_id) + return manager + + def get( + self, run_id: str, root: Path | str | None = None + ) -> SolutionPathEngine | None: + if root is not None: + lifecycle = self._runs.get(self._identity(root, run_id)) + return lifecycle.manager if lifecycle is not None else None + matches = [ + lifecycle.manager + for (_, candidate_run_id), lifecycle in self._runs.items() + if candidate_run_id == run_id and lifecycle.manager is not None + ] + if len(matches) > 1: + raise ValueError("run_id is ambiguous across solution path roots") + return matches[0] if matches else None + + def loaded_managers(self) -> tuple[SolutionPathEngine, ...]: + """Return process-loaded managers for fast read-only status snapshots.""" + return tuple( + lifecycle.manager + for lifecycle in self._runs.values() + if lifecycle.manager is not None + ) + + def latest_loaded_manager(self) -> SolutionPathEngine | None: + """Return the most recently acquired manager when it is still loaded.""" + if self._last_acquired_identity is None: + return None + lifecycle = self._runs.get(self._last_acquired_identity) + return lifecycle.manager if lifecycle is not None else None + + async def pause(self, run_id: str, root: Path | str | None = None) -> None: + lifecycles = ( + [await self._lifecycle(root, run_id)] + if root is not None + else await self._matching_lifecycles(run_id) + ) + if root is None and len(lifecycles) > 1: + raise ValueError("run_id is ambiguous across solution path roots") + for lifecycle in lifecycles: + async with lifecycle.lock: + if lifecycle.manager is not None: + lifecycle.generation += 1 + await lifecycle.manager.stop() + + async def clear(self, run_id: str, root: Path | str | None = None) -> None: + lifecycles = ( + [await self._lifecycle(root, run_id)] + if root is not None + else await self._matching_lifecycles(run_id) + ) + if root is None and len(lifecycles) > 1: + raise ValueError("run_id is ambiguous across solution path roots") + for lifecycle in lifecycles: + async with lifecycle.lock: + lifecycle.generation += 1 + manager = lifecycle.manager + lifecycle.manager = None + lifecycle.workflow_mode = None + lifecycle.prompt_hash = None + if self._last_acquired_identity == self._identity(lifecycle.root, run_id): + self._last_acquired_identity = None + if manager is not None: + await manager.clear() + + async def clear_run(self, root: Path | str, run_id: str) -> None: + lifecycle = await self._lifecycle(root, run_id) + async with lifecycle.lock: + lifecycle.generation += 1 + manager = lifecycle.manager + lifecycle.manager = None + lifecycle.workflow_mode = None + lifecycle.prompt_hash = None + if self._last_acquired_identity == self._identity(lifecycle.root, run_id): + self._last_acquired_identity = None + if manager is not None: + await manager.clear() + directory = lifecycle.root / run_id + if directory.exists(): + await asyncio.to_thread(shutil.rmtree, directory) + + async def clear_workflow(self, root: Path | str, workflow_mode: str) -> None: + root_path = Path(root) + if not root_path.exists(): + return + run_ids = [] + for state_path in root_path.glob("*/solution_path_state.json"): + try: + payload = json.loads(await asyncio.to_thread(state_path.read_text, encoding="utf-8")) + except (OSError, ValueError, TypeError): + continue + if payload.get("workflow_mode") == workflow_mode: + run_ids.append(state_path.parent.name) + for run_id in run_ids: + await self.clear_run(root_path, run_id) + + async def clear_manager(self, manager: SolutionPathEngine | None) -> None: + if manager is None: + return + matches = [ + lifecycle + for lifecycle in await self._matching_lifecycles(manager.run_id) + if lifecycle.manager is manager + ] + if len(matches) != 1: + if not matches: + return + raise ValueError("manager is registered under multiple solution path roots") + await self.clear(manager.run_id, matches[0].root) + + +solution_path_registry = SolutionPathManagerRegistry() diff --git a/backend/shared/solution_path/reviewer.py b/backend/shared/solution_path/reviewer.py new file mode 100644 index 0000000..e3d27e0 --- /dev/null +++ b/backend/shared/solution_path/reviewer.py @@ -0,0 +1,146 @@ +"""Shared production reviewer contract for Progressive Solution Path proposals.""" + +from __future__ import annotations + +import json +from collections.abc import Awaitable, Callable +from typing import Any + +from pydantic import ValidationError + +from backend.shared.json_parser import ( + parse_json, + sanitize_model_output_for_retry_context, +) +from backend.shared.utils import count_tokens + +from .models import PlanProposal, ReviewDecision, SolutionPlan + + +CompletionCall = Callable[[list[dict[str, str]]], Awaitable[Any]] + + +def build_review_prompt( + *, + user_prompt: str, + proposal: PlanProposal, + current_plan: SolutionPlan | None, + extra_context: str = "", +) -> str: + return ( + "You are production Main Submitter 1 reviewing a validator-proposed " + "Progressive Solution Path. Preserve the user's exact objective. Approve only " + "a material improvement. You may reject it, approve it as written, or perform " + "checked follow-up edits. Follow-up edits may replace `main_route`, replace the " + "whole `followup_route`, or use `edits` with add/update/delete/check operations " + "over stable step_id values. Use `more_edits=true` only when another review pass " + "is genuinely required; otherwise finish with approve or reject. Never treat " + "the plan as evidence.\n\n" + "Return one JSON object only:\n" + '{"decision":"approve|reject|followup","reasoning":"...",' + '"main_route":"optional replacement","followup_route":null,' + '"edits":[{"operation":"add|update|delete|check","step_id":"...",' + '"step":null,"after_step_id":null,"expected_title":null}],' + '"more_edits":false}\n\n' + f"USER PROMPT:\n{user_prompt}\n\n" + f"{extra_context}\n\n" + "CURRENT PLAN:\n" + f"{current_plan.model_dump_json(indent=2) if current_plan else 'None'}\n\n" + "PROPOSAL:\n" + f"{proposal.model_dump_json(indent=2)}" + ) + + +def compact_review_prompt( + *, + user_prompt: str, + proposal: PlanProposal, + current_plan: SolutionPlan | None, + extra_context: str = "", +) -> str: + """Drop durable bookkeeping fields while retaining the complete bounded route.""" + proposal_payload = { + "proposal_id": proposal.proposal_id, + "base_revision": proposal.base_revision, + "main_route": proposal.main_route, + "route": proposal.route.model_dump(mode="json"), + "rationale": proposal.rationale, + "feedback": proposal.feedback, + "source_phase": proposal.source_phase, + "source_decision": proposal.source_decision, + } + plan_payload = ( + { + "revision": current_plan.revision, + "main_route": current_plan.main_route, + "route": current_plan.route.model_dump(mode="json"), + } + if current_plan + else None + ) + return ( + "Review this bounded Progressive Solution Path update as Main Submitter 1. " + "Preserve the user's objective. Return one JSON object matching " + '{"decision":"approve|reject|followup","reasoning":"...",' + '"main_route":null,"followup_route":null,"edits":[],"more_edits":false}. ' + "Use followup only when another pass is genuinely needed.\n\n" + f"USER PROMPT:\n{user_prompt}\n\n" + f"{extra_context}\n\n" + f"CURRENT PLAN:\n{json.dumps(plan_payload, ensure_ascii=False)}\n\n" + f"PROPOSAL:\n{json.dumps(proposal_payload, ensure_ascii=False)}" + ) + + +async def review_with_json_retry( + *, + prompt: str, + call_completion: CompletionCall, + extract_text: Callable[[Any], str], + context_window: int, + max_output_tokens: int, + compact_prompt: str | None = None, +) -> ReviewDecision: + """Run one bounded sanitized JSON repair without replaying raw model output.""" + + max_input = max(1, context_window - max_output_tokens) + prompt_tokens = count_tokens(prompt) + if prompt_tokens > max_input: + if compact_prompt is not None and count_tokens(compact_prompt) <= max_input: + prompt = compact_prompt + prompt_tokens = count_tokens(prompt) + else: + raise ValueError( + "solution-path reviewer mandatory context exceeds the configured " + f"input budget ({prompt_tokens} > {max_input} tokens); automatic " + "bookkeeping compaction could not fit the complete bounded route" + ) + messages = [{"role": "user", "content": prompt}] + last_error = "invalid JSON response" + for attempt in range(2): + response = await call_completion(messages) + raw = extract_text(response) + try: + parsed = parse_json(raw) + if isinstance(parsed, list): + parsed = parsed[0] if parsed else {} + if not isinstance(parsed, dict): + raise ValueError("review response must be a JSON object") + return ReviewDecision.model_validate(parsed) + except (ValidationError, ValueError, TypeError, KeyError) as exc: + last_error = f"{type(exc).__name__}: {exc}" + if attempt: + break + safe = sanitize_model_output_for_retry_context(raw, max_chars=1500) + repair = ( + "Your prior response did not satisfy the required JSON schema. " + f"Validation error: {last_error}. Return one corrected JSON object only." + ) + retry_messages = [ + messages[0], + {"role": "assistant", "content": safe}, + {"role": "user", "content": repair}, + ] + if sum(count_tokens(item["content"]) for item in retry_messages) > max_input: + retry_messages = [messages[0], {"role": "user", "content": repair}] + messages = retry_messages + raise ValueError(f"solution-path reviewer JSON retry failed: {last_error}") diff --git a/backend/shared/syntheticlib4_client.py b/backend/shared/syntheticlib4_client.py index 4a457e7..6ef3537 100644 --- a/backend/shared/syntheticlib4_client.py +++ b/backend/shared/syntheticlib4_client.py @@ -1,8 +1,9 @@ """SyntheticLib4 corpus client used by the proof-search build slice. -The production SyntheticLib.com service is still under construction, so this -client implements the MOTO-side contract against offline/mock data while keeping -the same public methods that the live adapter will use later. +The production SyntheticLib.com service is still under construction. Normal +runtime clients therefore expose no corpus records until an authorized local +snapshot is explicitly activated. Tests may still construct a client with an +explicit fixture directory to exercise the future contract. """ from __future__ import annotations @@ -22,15 +23,6 @@ SYNTHETICLIB4_SCHEMA_VERSION = "syntheticlib4.mock_client.v1" _REPO_ROOT = Path(__file__).resolve().parents[2] _DEFAULT_FIXTURE_DIR = _REPO_ROOT / "tests" / "fixtures" / "syntheticlib4" -_BUILTIN_RELEASE_ID = "stable-2026-06-11" -_MOCK_SCOPES = [ - "proofs:read", - "releases:read", - "deltas:read", - "usage:write", - "account:status", - "user_proofs:read", -] class SyntheticLib4ClientError(RuntimeError): @@ -64,7 +56,7 @@ def get_status(self) -> dict[str, Any]: auth_mode = "local_snapshot" if source_kind == "data_root_snapshot" else "offline_fixture" else: status = self._builtin_account_status() - auth_mode = "built_in_offline_fixture" + auth_mode = "inactive" credential_configured = self.has_configured_credentials() return { @@ -115,7 +107,7 @@ def list_releases(self, channel: str | None = None) -> dict[str, Any]: requested_channel = (channel or manifest.get("channel") or "stable").strip() release_channel = str(manifest.get("channel") or "stable") releases = [] - if requested_channel == release_channel: + if manifest.get("release_id") and requested_channel == release_channel: releases.append( { "release_id": manifest.get("release_id", ""), @@ -139,19 +131,19 @@ def list_releases(self, channel: str | None = None) -> dict[str, Any]: } def get_release_manifest(self) -> dict[str, Any]: - """Load the local mock release manifest.""" + """Load the active local release manifest, or an empty pending manifest.""" source_dir, _source_kind = self._current_source_dir() manifest_path = source_dir / "release_manifest.json" if source_dir else None if manifest_path and manifest_path.exists(): return self._load_json(manifest_path) - return self._builtin_release_manifest() + return self._pending_release_manifest() def load_proof_metadata(self) -> list[dict[str, Any]]: - """Load SyntheticLib4 proof metadata JSONL fixture records.""" + """Load active SyntheticLib4 proof metadata, returning none when inactive.""" source_dir, _source_kind = self._current_source_dir() metadata_path = source_dir / "proof_metadata.jsonl" if source_dir else None if not metadata_path or not metadata_path.exists(): - return self._builtin_proof_metadata() + return [] records: list[dict[str, Any]] = [] for line_number, raw_line in enumerate(metadata_path.read_text(encoding="utf-8").splitlines(), 1): @@ -179,6 +171,18 @@ def validate_local_snapshot(self) -> dict[str, Any]: source_dir, source_kind = self._current_source_dir() manifest = self.get_release_manifest() records = self.load_proof_metadata() + if source_dir is None: + return { + "valid": False, + "contract_version": SYNTHETICLIB4_CONTRACT_VERSION, + "release_id": "", + "channel": manifest.get("channel", "stable"), + "proof_count": 0, + "fixture_source": source_kind, + "snapshot_dir": "", + "file_checks": [], + "reason": "SyntheticLib4 is not active; no local snapshot is installed.", + } required_manifest_fields = [ "contract_version", "schema_version", @@ -418,9 +422,12 @@ def _current_source_dir(self) -> tuple[Path | None, str]: data_snapshot = self._active_data_snapshot_dir() if data_snapshot is not None: return data_snapshot, "data_root_snapshot" - if (self.fixture_dir / "release_manifest.json").exists() or (self.fixture_dir / "proof_metadata.jsonl").exists(): + if self._fixture_dir_explicit and ( + (self.fixture_dir / "release_manifest.json").exists() + or (self.fixture_dir / "proof_metadata.jsonl").exists() + ): return self.fixture_dir, "filesystem" - return None, "built_in" + return None, "inactive" def _active_data_snapshot_dir(self, channel: str = "stable") -> Path | None: candidate = self.snapshot_root / "releases" / validate_single_path_component(channel, "SyntheticLib4 channel") @@ -434,7 +441,7 @@ def _manifest_uri(self) -> str: return f"file://{source_dir / 'release_manifest.json'}" if source_kind == "filesystem": return "fixture://syntheticlib4/release_manifest.json" - return "builtin://syntheticlib4/release_manifest.json" + return "" def _validate_snapshot_source_tree(self, source_path: Path) -> None: required = {"release_manifest.json", "proof_metadata.jsonl"} @@ -561,81 +568,35 @@ def _builtin_account_status(self) -> dict[str, Any]: return { "contract_version": SYNTHETICLIB4_CONTRACT_VERSION, "schema_version": "syntheticlib4.account_status.v1", - "authenticated": True, - "membership_active": True, - "membership_tier": "offline_mock", + "authenticated": False, + "membership_active": False, + "membership_tier": "", "access_expires_at": "", - "scopes": list(_MOCK_SCOPES), + "scopes": [], "quota": { - "api_requests_remaining_day": 2000, - "text_searches_remaining_month": 2000, - "semantic_searches_remaining_month": 200, + "api_requests_remaining_day": 0, + "text_searches_remaining_month": 0, + "semantic_searches_remaining_month": 0, }, } - def _builtin_release_manifest(self) -> dict[str, Any]: + def _pending_release_manifest(self) -> dict[str, Any]: return { "contract_version": SYNTHETICLIB4_CONTRACT_VERSION, "schema_version": "syntheticlib4.release_manifest.v1", - "release_id": _BUILTIN_RELEASE_ID, + "release_id": "", "channel": "stable", - "generated_at": "2026-06-11T00:00:00Z", - "lean_toolchain": "leanprover/lean4:v4.18.0", - "mathlib_revision": "mock-mathlib-rev", - "syntheticlib4_revision": "built-in-mock", - "license_terms_id": "syntheticlib4-member-license-v1", - "proof_count": 30, - "novelty_distribution": { - "novel_formalization": 20, - "novel_reformulation": 7, - "minor_mathematical_discovery": 3, - }, + "generated_at": "", + "lean_toolchain": "", + "mathlib_revision": "", + "syntheticlib4_revision": "", + "license_terms_id": "", + "proof_count": 0, + "novelty_distribution": {}, "compatible_moto_contract_versions": [SYNTHETICLIB4_CONTRACT_VERSION], "files": [], } - def _builtin_proof_metadata(self) -> list[dict[str, Any]]: - records: list[dict[str, Any]] = [] - for index in range(1, 31): - theorem_name = f"SyntheticLib4.Mock.builtin_helper_{index:03d}" - theorem_statement = f"theorem builtin_helper_{index:03d} : True" - lean_code = ( - "import Mathlib\n\n" - f"theorem builtin_helper_{index:03d} : True := by\n" - " trivial\n" - ) - fingerprint = f"sl4_builtin_fp_{index:03d}" - statement_hash = hashlib.sha256(theorem_statement.encode("utf-8")).hexdigest() - code_hash = hashlib.sha256(lean_code.encode("utf-8")).hexdigest() - metadata_only = index > 20 - records.append( - { - "fingerprint": fingerprint, - "display_title": f"Built-in SyntheticLib4 fixture proof {index}", - "theorem_name": theorem_name, - "theorem_statement": theorem_statement, - "informal_statement": "A built-in offline fixture proof for MOTO proof-search smoke tests.", - "proof_description": "Uses `trivial` to close a True goal.", - "theorem_statement_hash": statement_hash, - "lean_code": "" if metadata_only else lean_code, - "lean_code_hash": code_hash, - "imports": ["Mathlib"], - "dependency_names": ["True.intro"], - "topic_tags": ["fixture"], - "domain_tags": ["logic"], - "module": "SyntheticLib4.Mock", - "source_path": "SyntheticLib4/Mock.lean", - "line_range": {"start": index, "end": index + 2}, - "novelty_rank": "novel_formalization", - "novelty_confidence": 0.5, - "validation_record_id": f"builtin_val_{index:03d}", - "release_membership": "stable", - "license_terms_id": "syntheticlib4-member-license-v1", - "hydration_url": None, - } - ) - return records - syntheticlib4_client = SyntheticLib4Client() diff --git a/backend/shared/workflow_start_guard.py b/backend/shared/workflow_start_guard.py index 1b11437..27e6e61 100644 --- a/backend/shared/workflow_start_guard.py +++ b/backend/shared/workflow_start_guard.py @@ -5,19 +5,95 @@ import asyncio from contextlib import asynccontextmanager +from dataclasses import dataclass from typing import AsyncIterator +from backend.shared.sleep_inhibitor import sleep_inhibitor + + +@dataclass(frozen=True) +class WorkflowLease: + owner: str + generation: int + class WorkflowStartGuard: - """Serialize conflict checks and startup side effects across top-level modes.""" + """Serialize starts and retain the committed top-level workflow owner.""" def __init__(self) -> None: self._lock = asyncio.Lock() + self._active_lease: WorkflowLease | None = None + self._generation = 0 @asynccontextmanager async def reserve(self) -> AsyncIterator[None]: async with self._lock: yield + @property + def active_owner(self) -> str | None: + return self._active_lease.owner if self._active_lease else None + + @property + def active_lease(self) -> WorkflowLease | None: + return self._active_lease + + def commit(self, owner: str) -> WorkflowLease: + """Commit a successfully started top-level workflow.""" + if not owner: + raise ValueError("Workflow owner must be non-empty") + if self._active_lease and self._active_lease.owner == owner: + return self._active_lease + if self._active_lease is not None: + raise RuntimeError( + f"Top-level workflow '{self._active_lease.owner}' already owns execution" + ) + self._generation += 1 + lease = WorkflowLease(owner=owner, generation=self._generation) + self._active_lease = lease + try: + sleep_inhibitor.acquire(lease) + except Exception: + # Power inhibition is best-effort and must never fail workflow startup. + import logging + logging.getLogger(__name__).exception( + "Unable to request desktop sleep inhibition for %s", owner + ) + return lease + + def release(self, lease: WorkflowLease | str | None) -> bool: + """Release only the exact current lease. + + Owner-only releases are intentionally rejected: an asynchronous callback + from an older run must never release a newer generation of the same mode. + """ + if lease is None or self._active_lease is None: + return False + if isinstance(lease, str): + return False + if self._active_lease != lease: + return False + self._active_lease = None + try: + sleep_inhibitor.release(lease) + except Exception: + import logging + logging.getLogger(__name__).exception( + "Unable to release desktop sleep inhibition for %s", lease.owner + ) + return True + + def release_all(self) -> None: + """Clear logical ownership and desktop sleep inhibition at shutdown.""" + self._generation += 1 + self._active_lease = None + try: + sleep_inhibitor.release_all() + except Exception: + import logging + logging.getLogger(__name__).exception( + "Unable to release all desktop sleep inhibition" + ) + workflow_start_guard = WorkflowStartGuard() diff --git a/frontend/.npmrc b/frontend/.npmrc new file mode 100644 index 0000000..faeb8ab --- /dev/null +++ b/frontend/.npmrc @@ -0,0 +1,3 @@ +save-exact=true +fund=false +audit=true diff --git a/frontend/package-lock.json b/frontend/package-lock.json index 6f893ce..eec97f2 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -1,20 +1,21 @@ { "name": "asi-aggregator-frontend", - "version": "1.1.02", + "version": "1.1.03", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "asi-aggregator-frontend", - "version": "1.1.02", + "version": "1.1.03", "license": "MIT", "dependencies": { - "dompurify": "^3.2.4", + "dompurify": "^3.4.12", "katex": "^0.16.27", "react": "^18.2.0", "react-dom": "^18.2.0" }, "devDependencies": { + "@playwright/test": "^1.54.1", "@testing-library/jest-dom": "^6.9.1", "@testing-library/react": "^16.3.2", "@testing-library/user-event": "^14.6.1", @@ -830,6 +831,22 @@ "url": "https://github.com/sponsors/Boshen" } }, + "node_modules/@playwright/test": { + "version": "1.61.1", + "resolved": "https://registry.npmjs.org/@playwright/test/-/test-1.61.1.tgz", + "integrity": "sha512-8nKv6+0RJSL9FE4jYOEGXnPeM/Hg12qZpmqzZjRh3qM0Y7c3z1mrOTfFLids72RDQYVh9WpLEfR5WdpNX4fkig==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "playwright": "1.61.1" + }, + "bin": { + "playwright": "cli.js" + }, + "engines": { + "node": ">=18" + } + }, "node_modules/@rolldown/binding-android-arm64": { "version": "1.0.3", "resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.3.tgz", @@ -1568,9 +1585,9 @@ "peer": true }, "node_modules/dompurify": { - "version": "3.4.11", - "resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.4.11.tgz", - "integrity": "sha512-zhlUV12GsaRzMsf9q5M254YhA4+VuF0fG+QFqu6aYpoGlKtz+w8//jBcGVYBgQkR5GHjUomejY84AV+/uPbWdw==", + "version": "3.4.12", + "resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.4.12.tgz", + "integrity": "sha512-zQvGet8Z2sWbQhCmfFz/T5QWH2oBmjnqK3qvOjaqaNLrLEF912WamU+ohnTp0TCep/MFVHpdJuCZEdFOdTnEFg==", "license": "(MPL-2.0 OR Apache-2.0)", "optionalDependencies": { "@types/trusted-types": "^2.0.7" @@ -2180,6 +2197,53 @@ "url": "https://github.com/sponsors/jonschlinkert" } }, + "node_modules/playwright": { + "version": "1.61.1", + "resolved": "https://registry.npmjs.org/playwright/-/playwright-1.61.1.tgz", + "integrity": "sha512-DWnY5o3YbLWK4GovuAVwpqL+1VwGNdUGrRr++8j8PtQQzvAVZUIMjKQ90fY689sEJZJBbZVw1rXaOKSTitkzPQ==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "playwright-core": "1.61.1" + }, + "bin": { + "playwright": "cli.js" + }, + "engines": { + "node": ">=18" + }, + "optionalDependencies": { + "fsevents": "2.3.2" + } + }, + "node_modules/playwright-core": { + "version": "1.61.1", + "resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.61.1.tgz", + "integrity": "sha512-h7Qlt6m4REp25qvIdvbDtVmD4LqVXfpRxhORv9L0jzETM05p4fuPJ3dKyuSXQxDSbXnmS79HAgi9589lGSpLkg==", + "dev": true, + "license": "Apache-2.0", + "bin": { + "playwright-core": "cli.js" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/playwright/node_modules/fsevents": { + "version": "2.3.2", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.2.tgz", + "integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^8.16.0 || ^10.6.0 || >=11.0.0" + } + }, "node_modules/postcss": { "version": "8.5.15", "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.15.tgz", diff --git a/frontend/package.json b/frontend/package.json index aa028fa..96000df 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -1,7 +1,7 @@ { "name": "asi-aggregator-frontend", - "version": "1.1.02", - "description": "Frontend UI for MOTO S.T.E.M. Mathematics Variant - Autonomous ASI Research System for Novel S.T.E.M. Mathematical Paper Generation", + "version": "1.1.03", + "description": "Frontend UI for MOTO Autonomous ASI multi-agent research and rigorous solution generation with optional Lean 4 proof verification", "author": "Intrafere LLC", "license": "MIT", "type": "module", @@ -10,10 +10,11 @@ "build": "vite build", "preview": "vite preview", "test": "vitest run", - "test:watch": "vitest" + "test:watch": "vitest", + "test:browser": "playwright test" }, "dependencies": { - "dompurify": "^3.2.4", + "dompurify": "^3.4.12", "katex": "^0.16.27", "react": "^18.2.0", "react-dom": "^18.2.0" @@ -22,6 +23,7 @@ "@testing-library/jest-dom": "^6.9.1", "@testing-library/react": "^16.3.2", "@testing-library/user-event": "^14.6.1", + "@playwright/test": "^1.54.1", "@types/react": "^18.2.43", "@types/react-dom": "^18.2.17", "@vitejs/plugin-react": "^6.0.2", diff --git a/frontend/playwright.config.js b/frontend/playwright.config.js new file mode 100644 index 0000000..ab4c06f --- /dev/null +++ b/frontend/playwright.config.js @@ -0,0 +1,36 @@ +import { defineConfig, devices } from '@playwright/test'; + +const port = Number(process.env.MOTO_BROWSER_TEST_PORT || 4173); +const baseURL = `http://127.0.0.1:${port}`; + +export default defineConfig({ + testDir: '../tests/workflow_browser_smoke', + fullyParallel: false, + workers: 1, + forbidOnly: Boolean(process.env.CI), + retries: process.env.CI ? 1 : 0, + reporter: process.env.CI + ? [['line'], ['html', { outputFolder: '../playwright-report', open: 'never' }]] + : 'line', + outputDir: '../test-results/playwright', + use: { + baseURL, + trace: 'retain-on-failure', + screenshot: 'only-on-failure', + video: 'retain-on-failure', + serviceWorkers: 'block', + }, + projects: [ + { + name: 'chromium', + use: { ...devices['Desktop Chrome'] }, + }, + ], + webServer: { + command: `npm run preview -- --host 127.0.0.1 --port ${port} --strictPort`, + url: baseURL, + reuseExistingServer: !process.env.CI, + stdout: 'pipe', + stderr: 'pipe', + }, +}); diff --git a/frontend/src/App.jsx b/frontend/src/App.jsx index 7805462..d04a695 100644 --- a/frontend/src/App.jsx +++ b/frontend/src/App.jsx @@ -29,6 +29,8 @@ import { LeanOJSettings, } from './components/leanoj'; import WorkflowPanel from './components/WorkflowPanel'; +import SolutionPathModal from './components/SolutionPathModal'; +import { solutionPathEventMatches } from './utils/solutionPathPresentation'; import BoostControlModal from './components/BoostControlModal'; import StartupProviderSetupModal from './components/StartupProviderSetupModal'; import OpenRouterApiKeyModal from './components/OpenRouterApiKeyModal'; @@ -44,8 +46,18 @@ import CreditExhaustionNotificationStack from './components/CreditExhaustionNoti import CodexOAuthNotificationStack from './components/CodexOAuthNotificationStack'; import UpdateNotificationBanner from './components/UpdateNotificationBanner'; import PaperCritiqueModal from './components/PaperCritiqueModal'; +import intrafereLogoMark from './assets/brand/intrafere-logo-no-text.png'; import { websocket } from './services/websocket'; -import { api, autonomousAPI, cloudAccessAPI, compilerAPI, connectivityAPI, leanojAPI, openRouterAPI } from './services/api'; +import { + API_ERROR_KINDS, + api, + autonomousAPI, + cloudAccessAPI, + compilerAPI, + connectivityAPI, + leanojAPI, + openRouterAPI, +} from './services/api'; import { LM_STUDIO_STARTUP_CHOICE, RECOMMENDED_PROFILE_KEY, @@ -68,11 +80,13 @@ import { CLOUD_ACCESS_PROVIDERS, isCloudAccessProvider } from './utils/oauthProv import { formatContextOverflowActivityMessage, formatAssistantProofPackEventMessage, + formatSolutionPathEventMessage, buildAutonomousProofProviderPauseActivity, buildRejectionFeedbackNoticeActivity, hasRecentAssistantProofPackDuplicate, shouldAddRejectionFeedbackNotice, } from './utils/activityStyles'; +import { sanitizePersistedActivityValue } from './utils/activityPersistence'; import { canStorePromptDraftInLocalStorage, readPromptDraftSync, @@ -95,9 +109,12 @@ const EMBEDDING_MODEL_HINTS = ['embed', 'embedding', 'nomic', 'bge', 'e5', 'gte' const AUTONOMOUS_ROLE_PREFIXES = ['validator', 'assistant', 'writer', 'high_param']; const HIGH_SCORE_CRITIQUE_THRESHOLD = 6.25; const SEEN_HIGH_SCORE_CRITIQUES_STORAGE_KEY = 'seenHighScoreCritiqueNotifications'; -const DISMISSED_OAUTH_PROVIDER_NOTIFICATIONS_STORAGE_KEY = 'dismissedOAuthProviderNotifications'; +// Read-only compatibility key. Older releases stored opaque notification IDs +// here, while an intermediate release stored SHA-256 fingerprints. +const DISMISSED_PROVIDER_NOTIFICATION_IDS_STORAGE_KEY = 'dismissedOAuthProviderNotifications'; +const DISMISSED_PROVIDER_NOTIFICATION_FINGERPRINT_PREFIX = 'dismissedOAuthProviderNotificationFingerprint:'; const MAX_SEEN_HIGH_SCORE_CRITIQUES = 500; -const MAX_DISMISSED_OAUTH_PROVIDER_NOTIFICATIONS = 500; +const MAX_DISMISSED_PROVIDER_NOTIFICATION_IDS = 500; const MAX_LIVE_ACTIVITY_EVENTS = 5000; const AUTONOMOUS_LIVE_ACTIVITY_STORAGE_KEY = 'autonomous_live_activity_events'; const LEANOJ_LIVE_ACTIVITY_STORAGE_KEY = 'leanoj_live_activity_events'; @@ -187,31 +204,77 @@ function persistSeenHighScoreCritiques(seenSet) { } } -function readDismissedOAuthProviderNotifications() { +async function fingerprintProviderNotificationId(value) { + const input = new TextEncoder().encode(String(value || '')); + const digest = await window.crypto.subtle.digest('SHA-256', input); + return Array.from(new Uint8Array(digest), byte => byte.toString(16).padStart(2, '0')).join(''); +} + +function readLegacyDismissedProviderNotificationIds() { if (typeof window === 'undefined') { return new Set(); } try { - const raw = window.localStorage.getItem(DISMISSED_OAUTH_PROVIDER_NOTIFICATIONS_STORAGE_KEY); + const raw = window.localStorage.getItem(DISMISSED_PROVIDER_NOTIFICATION_IDS_STORAGE_KEY); const values = raw ? JSON.parse(raw) : []; return new Set(Array.isArray(values) ? values.filter(value => typeof value === 'string') : []); } catch (error) { - console.warn('Could not read dismissed OAuth provider notifications:', error); + console.warn('Could not read legacy dismissed provider notification IDs:', error); return new Set(); } } -function persistDismissedOAuthProviderNotifications(seenSet) { +function dismissedProviderNotificationFingerprintKey(fingerprint) { + return `${DISMISSED_PROVIDER_NOTIFICATION_FINGERPRINT_PREFIX}${fingerprint}`; +} + +function trimDismissedProviderNotificationMarkers() { + const markers = []; + for (let index = 0; index < window.localStorage.length; index += 1) { + const key = window.localStorage.key(index); + if (key?.startsWith(DISMISSED_PROVIDER_NOTIFICATION_FINGERPRINT_PREFIX)) { + markers.push({ + key, + createdAt: Number(window.localStorage.getItem(key)) || 0, + }); + } + } + markers + .sort((left, right) => left.createdAt - right.createdAt) + .slice(0, Math.max(0, markers.length - MAX_DISMISSED_PROVIDER_NOTIFICATION_IDS)) + .forEach(({ key }) => window.localStorage.removeItem(key)); +} + +export async function isProviderNotificationDismissed(notificationId) { + if (typeof window === 'undefined') { + return false; + } + + const normalizedId = String(notificationId || ''); + const fingerprint = await fingerprintProviderNotificationId(normalizedId); + if (window.localStorage.getItem(dismissedProviderNotificationFingerprintKey(fingerprint)) !== null) { + return true; + } + + const legacyIds = readLegacyDismissedProviderNotificationIds(); + return legacyIds.has(normalizedId) || legacyIds.has(fingerprint); +} + +export async function persistDismissedProviderNotificationId(notificationId) { if (typeof window === 'undefined') { return; } try { - const values = Array.from(seenSet).slice(-MAX_DISMISSED_OAUTH_PROVIDER_NOTIFICATIONS); - window.localStorage.setItem(DISMISSED_OAUTH_PROVIDER_NOTIFICATIONS_STORAGE_KEY, JSON.stringify(values)); + const fingerprint = await fingerprintProviderNotificationId(notificationId); + window.localStorage.setItem( + dismissedProviderNotificationFingerprintKey(fingerprint), + String(Date.now()), + ); + trimDismissedProviderNotificationMarkers(); } catch (error) { - console.warn('Could not save dismissed OAuth provider notifications:', error); + console.warn('Could not save dismissed provider notification IDs:', error); } } @@ -247,13 +310,9 @@ function buildOAuthProviderNotification(data = {}, fallbackProvider = 'oauth') { }; } -function addOAuthProviderNotification(setNotifications, data = {}, fallbackProvider = 'oauth') { +async function addOAuthProviderNotification(setNotifications, data = {}, fallbackProvider = 'oauth') { const notification = buildOAuthProviderNotification(data, fallbackProvider); - const dismissedNotifications = readDismissedOAuthProviderNotifications(); - if ( - dismissedNotifications.has(notification.notification_key) - || (data.id && dismissedNotifications.has(String(data.id))) - ) { + if (await isProviderNotificationDismissed(notification.notification_key)) { return; } setNotifications(prev => { @@ -362,7 +421,9 @@ function buildAggregatorConfigFromStorage() { assistantMaxOutput: settings.assistantMaxOutput ?? legacy.assistantMaxOutput ?? settings.validatorMaxOutput ?? legacy.validatorMaxOutput ?? DEFAULT_MAX_OUTPUT_TOKENS, assistantSuperchargeEnabled: Boolean(settings.assistantSuperchargeEnabled ?? legacy.assistantSuperchargeEnabled), creativityEmphasisBoostEnabled: Boolean(settings.creativityEmphasisBoostEnabled ?? legacy.creativityEmphasisBoostEnabled), - uploadedFiles: [], + uploadedFiles: Array.isArray(settings.uploadedFiles) + ? settings.uploadedFiles + : (Array.isArray(legacy.uploadedFiles) ? legacy.uploadedFiles : []), }; } @@ -513,15 +574,28 @@ function coercePositiveIntegerSetting(value, fallback) { return fallback; } -function readPersistedLiveActivity(storageKey) { +export function readPersistedLiveActivity(storageKey) { try { const savedEvents = localStorage.getItem(storageKey); if (!savedEvents) { return []; } - const parsed = JSON.parse(savedEvents); + const parsed = sanitizePersistedActivityValue(JSON.parse(savedEvents)); return Array.isArray(parsed) - ? parsed.filter((event) => event && typeof event === 'object').slice(-MAX_LIVE_ACTIVITY_EVENTS) + ? parsed + .filter((event) => event && typeof event === 'object') + .map((event) => { + const eventName = event.event || event.type; + const isOverflow = eventName === 'context_overflow_error' + || ( + (eventName === 'auto_research_stopped' || eventName === 'leanoj_stopped') + && event?.data?.reason === 'context_overflow' + ); + return isOverflow + ? { ...event, message: formatContextOverflowActivityMessage(event.data || {}) } + : event; + }) + .slice(-MAX_LIVE_ACTIVITY_EVENTS) : []; } catch (error) { console.error(`Failed to load ${storageKey}:`, error); @@ -529,6 +603,13 @@ function readPersistedLiveActivity(storageKey) { } } +export function shouldRecordWorkflowStoppedActivity(eventName, data = {}) { + return !( + (eventName === 'auto_research_stopped' || eventName === 'leanoj_stopped') + && data?.reason === 'context_overflow' + ); +} + function compactPersistedActivityValue(value, depth = 0) { if (value == null || typeof value === 'number' || typeof value === 'boolean') { return value; @@ -556,19 +637,22 @@ function compactPersistedActivityValue(value, depth = 0) { return String(value); } -function compactLiveActivityEvent(event) { +export function compactLiveActivityEvent(event) { if (!event || typeof event !== 'object') { return null; } + const sanitizedEvent = sanitizePersistedActivityValue(event); return { - event: event.event || event.type || '', - type: event.type, - timestamp: event.timestamp || event.fullTimestamp || '', - fullTimestamp: event.fullTimestamp, - message: typeof event.message === 'string' - ? compactPersistedActivityValue(event.message) + event: sanitizedEvent.event || sanitizedEvent.type || '', + type: sanitizedEvent.type, + timestamp: sanitizedEvent.timestamp || sanitizedEvent.fullTimestamp || '', + fullTimestamp: sanitizedEvent.fullTimestamp, + // Persist the user-visible message after recursive credential redaction so + // live activity remains useful across reloads without storing secrets. + message: typeof sanitizedEvent.message === 'string' + ? compactPersistedActivityValue(sanitizedEvent.message) : '', - data: compactPersistedActivityValue(event.data || {}), + data: compactPersistedActivityValue(sanitizedEvent.data || {}), }; } @@ -658,6 +742,8 @@ function App() { // Track if any workflow is running (for WorkflowPanel visibility) const [anyWorkflowRunning, setAnyWorkflowRunning] = useState(false); + const [solutionPathSnapshot, setSolutionPathSnapshot] = useState(null); + const solutionPathFenceRef = useRef(null); // Track WorkflowPanel collapse state for sliding boost buttons const [workflowPanelCollapsed, setWorkflowPanelCollapsed] = useState(() => { @@ -814,6 +900,7 @@ function App() { assistantMaxOutput: config.assistantMaxOutput, assistantSuperchargeEnabled: config.assistantSuperchargeEnabled, creativityEmphasisBoostEnabled: config.creativityEmphasisBoostEnabled, + uploadedFiles: config.uploadedFiles || [], }; try { // Save to both old and new keys. @@ -822,10 +909,11 @@ function App() { } catch (error) { console.warn('Could not persist Aggregator settings to localStorage:', error); } - }, [config.userPrompt, config.submitterConfigs, config.validatorModel, config.validatorProvider, config.validatorOpenrouterProvider, config.validatorOpenrouterReasoningEffort, config.validatorLmStudioFallback, config.validatorContextSize, config.validatorMaxOutput, config.validatorSuperchargeEnabled, config.assistantModel, config.assistantProvider, config.assistantOpenrouterProvider, config.assistantOpenrouterReasoningEffort, config.assistantLmStudioFallback, config.assistantContextSize, config.assistantMaxOutput, config.assistantSuperchargeEnabled, config.creativityEmphasisBoostEnabled]); + }, [config.userPrompt, config.submitterConfigs, config.validatorModel, config.validatorProvider, config.validatorOpenrouterProvider, config.validatorOpenrouterReasoningEffort, config.validatorLmStudioFallback, config.validatorContextSize, config.validatorMaxOutput, config.validatorSuperchargeEnabled, config.assistantModel, config.assistantProvider, config.assistantOpenrouterProvider, config.assistantOpenrouterReasoningEffort, config.assistantLmStudioFallback, config.assistantContextSize, config.assistantMaxOutput, config.assistantSuperchargeEnabled, config.creativityEmphasisBoostEnabled, config.uploadedFiles]); // Autonomous mode state const [autonomousRunning, setAutonomousRunning] = useState(false); + const [autonomousStarting, setAutonomousStarting] = useState(false); const [autonomousStopping, setAutonomousStopping] = useState(false); const [autonomousStatus, setAutonomousStatus] = useState(null); const [autonomousActivity, setAutonomousActivity] = useState(() => ( @@ -864,6 +952,7 @@ function App() { const [proofNotifications, setProofNotifications] = useState([]); const [selectedProofId, setSelectedProofId] = useState(null); const [selectedProofScope, setSelectedProofScope] = useState('autonomous'); + const [selectedHistoryProof, setSelectedHistoryProof] = useState(null); const [proofRefreshToken, setProofRefreshToken] = useState(0); const [latestProofDependencyEvent, setLatestProofDependencyEvent] = useState(null); @@ -882,6 +971,8 @@ function App() { // Live refs used by websocket listeners (which are registered once) const autonomousRunningRef = useRef(autonomousRunning); const autonomousTierRef = useRef(autonomousStatus?.current_tier || null); + const autonomousLifecycleGenerationRef = useRef(0); + const leanojLifecycleGenerationRef = useRef(0); const openRouterKeyJustSavedRef = useRef(false); const cloudAccessJustConfiguredRef = useRef(false); const seenHighScoreCritiquesRef = useRef(null); @@ -1515,6 +1606,13 @@ function App() { if (round <= 0 || maxRounds <= 1) return ''; return `Proof round ${round}/${maxRounds}`; }; + const formatProofCandidatesFoundMessage = (data = {}) => { + const count = Number(data.count || 0); + const roundLabel = proofRoundLabel(data); + const subject = count === 1 ? 'proof candidate' : 'proof candidates'; + const prefix = roundLabel ? `${roundLabel} discovery` : 'Proof discovery'; + return `${prefix} found ${count} ${subject}; ${count} will be attempted`; + }; const proofLeanResponse = (data = {}) => { if (data.lean_response) { let response = data.lean_response; @@ -1640,6 +1738,36 @@ function App() { data }); })); + + [ + 'solution_path_activated', + 'solution_path_proposal_queued', + 'solution_path_proposal_reviewing', + 'solution_path_updated', + 'solution_path_proposal_rejected', + 'solution_path_proposal_retry_queued', + 'solution_path_proposal_user_repair_required', + 'solution_path_proposal_resumed', + ].forEach((eventName) => { + unsubscribers.push(websocket.on(eventName, (data = {}) => { + const expectedMode = String(data.workflow_mode || data.mode || ''); + if ( + solutionPathFenceRef.current + && !solutionPathEventMatches(data, solutionPathFenceRef.current, expectedMode) + ) return; + const activityEvent = { + event: eventName, + message: formatSolutionPathEventMessage(eventName, data), + timestamp: new Date().toISOString(), + ...data, + }; + if (data.workflow_mode === 'leanoj') { + setLeanojActivity((prev) => [...prev, activityEvent].slice(-MAX_LIVE_ACTIVITY_EVENTS)); + } else if (data.workflow_mode === 'autonomous') { + setAutonomousActivity((prev) => [...prev, activityEvent].slice(-MAX_LIVE_ACTIVITY_EVENTS)); + } + })); + }); unsubscribers.push(websocket.on('paper_title_exploration_progress', (data) => { addActivity({ @@ -1910,10 +2038,26 @@ function App() { unsubscribers.push(websocket.on('proof_check_no_candidates', (data) => { setProofRefreshToken((prev) => prev + 1); + if (isLeanOJProofEvent(data) || isManualProofEvent(data)) return; + const roundLabel = proofRoundLabel(data); + addActivity({ + event: 'proof_check_no_candidates', + timestamp: getTimestamp(data), + message: `${roundLabel ? `${roundLabel} discovery` : 'Proof discovery'} found 0 proof candidates; no proofs will be attempted`, + data + }); })); unsubscribers.push(websocket.on('proof_check_candidates_found', (data) => { setProofRefreshToken((prev) => prev + 1); + if (isLeanOJProofEvent(data) || isManualProofEvent(data)) return; + if (data.source_type === 'compiler_rigor' && !isAutonomousTier2Active()) return; + addActivity({ + event: 'proof_check_candidates_found', + timestamp: getTimestamp(data), + message: formatProofCandidatesFoundMessage(data), + data + }); })); unsubscribers.push(websocket.on('proof_attempt_started', (data) => { @@ -1950,6 +2094,16 @@ function App() { setProofRefreshToken((prev) => prev + 1); })); + unsubscribers.push(websocket.on('proof_context_overflow', (data) => { + if (isLeanOJProofEvent(data) || isManualProofEvent(data)) return; + addActivity({ + event: 'proof_context_overflow', + timestamp: getTimestamp(data), + message: formatContextOverflowActivityMessage(data), + data + }); + })); + unsubscribers.push(websocket.on('proof_lean_accepted', (data) => { if (isLeanOJProofEvent(data) || isManualProofEvent(data)) return; addActivity({ @@ -2066,6 +2220,7 @@ function App() { })); unsubscribers.push(websocket.on('auto_research_started', (data = {}) => { + setAutonomousStarting(false); setAutonomousRunning(true); setAnyWorkflowRunning(true); setAutonomousStopping(false); @@ -2080,6 +2235,7 @@ function App() { unsubscribers.push(websocket.on('auto_research_resumed', (data) => { // Handle resume after crash/restart - sync running state console.log('Autonomous research resumed:', data); + setAutonomousStarting(false); setAutonomousRunning(true); setAnyWorkflowRunning(true); setAutonomousStopping(false); @@ -2099,16 +2255,20 @@ function App() { })); unsubscribers.push(websocket.on('auto_research_stopped', (data = {}) => { + autonomousLifecycleGenerationRef.current += 1; + setAutonomousStarting(false); setAutonomousRunning(false); setAutonomousStopping(false); setAnyWorkflowRunning(false); autonomousTierRef.current = null; - addActivity({ - event: 'auto_research_stopped', - timestamp: getTimestamp(data), - message: data.message || `Research stopped. Total: ${data.final_stats?.total_papers_completed || 0} papers`, - data - }); + if (shouldRecordWorkflowStoppedActivity('auto_research_stopped', data)) { + addActivity({ + event: 'auto_research_stopped', + timestamp: getTimestamp(data), + message: data.message || `Research stopped. Total: ${data.final_stats?.total_papers_completed || 0} papers`, + data + }); + } })); // Tier 3 events @@ -2351,7 +2511,7 @@ function App() { }; if (workflowMode === 'leanoj' || roleId.startsWith('leanoj_')) { addLeanOJActivityFromGlobalAlert(event); - } else if (autonomousRunningRef.current || shouldAddHungAlertToAutonomousFeed(data)) { + } else if (workflowMode === 'autonomous' || shouldAddHungAlertToAutonomousFeed(data)) { addActivity(event); } })); @@ -2836,9 +2996,16 @@ function App() { addLeanOJActivity('assistant_proof_pack_failed', data, formatAssistantProofPackEventMessage('assistant_proof_pack_failed', data)); }], ['leanoj_stopped', (data) => { + leanojLifecycleGenerationRef.current += 1; setLeanojRunning(false); setAnyWorkflowRunning(false); - addLeanOJActivity('leanoj_stopped', data, data?.message || 'Proof Solver stopped'); + if (shouldRecordWorkflowStoppedActivity('leanoj_stopped', data)) { + addLeanOJActivity( + 'leanoj_stopped', + data, + data?.message || 'Proof Solver stopped' + ); + } leanojAPI.getStatus().then(setLeanojStatus).catch(console.error); }], ['leanoj_status_updated', (data) => setLeanojStatus(data)], @@ -2996,6 +3163,7 @@ function App() { // Autonomous handlers const handleAutonomousStart = async (researchPrompt) => { + const startGeneration = autonomousLifecycleGenerationRef.current; try { const lmStudioEnabled = capabilities.lmStudioEnabled; const superchargeAllowed = developerModeEnabled; @@ -3107,11 +3275,16 @@ function App() { allow_research_papers: autonomousConfig.allow_research_papers ?? true, tier3_enabled: autonomousConfig.tier3_enabled ?? false }); + if (startGeneration !== autonomousLifecycleGenerationRef.current) { + return false; + } setAutonomousRunning(true); + setAutonomousStarting(false); setAutonomousStopping(false); setAnyWorkflowRunning(true); return true; } catch (error) { + setAutonomousStarting(false); alert(`Failed to start autonomous research: ${error.details || error.message}`); return false; } @@ -3125,11 +3298,27 @@ function App() { setAutonomousStopping(true); try { await autonomousAPI.stop(); + autonomousLifecycleGenerationRef.current += 1; setAutonomousRunning(false); setAnyWorkflowRunning(false); - const status = await autonomousAPI.getStatus(); - setAutonomousStatus(status); + autonomousAPI.getStatus().then(setAutonomousStatus).catch((error) => { + console.warn('Autonomous research stopped, but status refresh failed:', error); + }); } catch (error) { + if (error.kind === API_ERROR_KINDS.AMBIGUOUS_TRANSPORT) { + try { + const status = await autonomousAPI.getStatus(); + setAutonomousStatus(status); + setAutonomousRunning(Boolean(status.is_running)); + setAnyWorkflowRunning(Boolean(status.is_running)); + if (!status.is_running) { + autonomousLifecycleGenerationRef.current += 1; + return; + } + } catch (statusError) { + console.error('Could not reconcile indeterminate autonomous stop:', statusError); + } + } alert(`Failed to stop autonomous research: ${error.message}`); } finally { setAutonomousStopping(false); @@ -3198,11 +3387,16 @@ function App() { }); const handleLeanOJStart = async (request) => { + const startGeneration = leanojLifecycleGenerationRef.current; try { await leanojAPI.start(normalizeLeanOJRequestForCapabilities(request)); + if (startGeneration !== leanojLifecycleGenerationRef.current) { + return; + } setLeanojRunning(true); - const status = await leanojAPI.getStatus(); - setLeanojStatus(status); + leanojAPI.getStatus().then(setLeanojStatus).catch((error) => { + console.warn('Proof Solver started, but status refresh failed:', error); + }); setLeanojProofRefreshToken((prev) => prev + 1); setAnyWorkflowRunning(true); } catch (error) { @@ -3213,11 +3407,27 @@ function App() { const handleLeanOJStop = async () => { try { await leanojAPI.stop(); + leanojLifecycleGenerationRef.current += 1; setLeanojRunning(false); setAnyWorkflowRunning(false); - const status = await leanojAPI.getStatus(); - setLeanojStatus(status); + leanojAPI.getStatus().then(setLeanojStatus).catch((error) => { + console.warn('Proof Solver stopped, but status refresh failed:', error); + }); } catch (error) { + if (error.kind === API_ERROR_KINDS.AMBIGUOUS_TRANSPORT) { + try { + const status = await leanojAPI.getStatus(); + setLeanojStatus(status); + setLeanojRunning(Boolean(status.is_running)); + setAnyWorkflowRunning(Boolean(status.is_running)); + if (!status.is_running) { + leanojLifecycleGenerationRef.current += 1; + return; + } + } catch (statusError) { + console.error('Could not reconcile indeterminate Proof Solver stop:', statusError); + } + } alert(`Failed to stop Proof Solver: ${error.message}`); } }; @@ -3344,6 +3554,34 @@ function App() { handleAutonomousTabSelect('auto-proofs'); }; + const handleOpenAssistantProof = (support = {}) => { + const corpus = String(support.corpus || '').toLowerCase(); + const proofId = support.proof_id || support.search_id || null; + const nextSelection = { + corpus, + proofId, + sessionId: support.session_id || '', + runId: support.run_id || '', + }; + setSelectedHistoryProof(nextSelection); + if (proofId) { + setSelectedProofId(proofId); + setSelectedProofScope(corpus === 'manual' ? 'manual' : 'autonomous'); + } + if (corpus === 'manual') { + handleManualTabSelect('compiler-proof-history'); + return; + } + if (corpus === 'leanoj') { + handleLeanOJTabSelect('leanoj-completed-proof-works'); + return; + } + if (corpus === 'moto') { + handleAutonomousTabSelect('auto-completed-works'); + setCompletedWorksSubTab('proof-library'); + } + }; + const handleModeChange = (nextMode) => { setAppMode(nextMode); }; @@ -3378,9 +3616,7 @@ function App() { setCodexOAuthNotifications(prev => { const notification = prev.find(n => n.id === notificationId); if (notification?.notification_key) { - const dismissed = readDismissedOAuthProviderNotifications(); - dismissed.add(notification.notification_key); - persistDismissedOAuthProviderNotifications(dismissed); + void persistDismissedProviderNotificationId(notification.notification_key); } return prev.filter(n => n.id !== notificationId); }); @@ -3665,7 +3901,7 @@ function App() { ]; const autonomousSettingsTabs = [ - { id: 'auto-completed-works', label: 'Your Completed Works Library', group: 'autonomous-settings' }, + { id: 'auto-completed-works', label: 'Completed Works Library', group: 'autonomous-settings' }, { id: 'auto-logs', label: 'API Call Logs', group: 'autonomous-settings' }, { id: 'auto-settings', label: 'Autonomous Model Selection & Settings', group: 'autonomous-settings' }, ]; @@ -3678,6 +3914,7 @@ function App() { { id: 'compiler-interface', label: 'Compiler', subtext: 'Part 2', subtextClass: 'green', group: 'compiler' }, { id: 'compiler-live-paper', label: 'Live Paper', subtext: 'Part 2 Live Results', subtextClass: 'green', group: 'compiler' }, { id: 'compiler-proofs', label: 'Mathematical Proofs', subtext: 'Manual Proofs', subtextClass: 'green', group: 'compiler' }, + { id: 'compiler-proof-history', label: 'Completed Proof Works', subtext: 'Manual History', group: 'compiler' }, { id: 'compiler-logs', label: 'Compiler Logs', group: 'compiler' }, { id: 'compiler-settings', label: 'Compiler Settings', group: 'compiler' }, ]; @@ -3704,18 +3941,23 @@ function App() { // Check if any workflow is running useEffect(() => { const checkWorkflowStatus = async () => { - try { - const [aggStatus, compStatus, autoStatus, leanojCurrentStatus] = await Promise.all([ - api.get('/api/aggregator/status').catch(() => ({ is_running: false })), - api.get('/api/compiler/status').catch(() => ({ is_running: false })), - autonomousAPI.getStatus().catch(() => ({ is_running: false })), - leanojAPI.getStatus().catch(() => ({ is_running: false })) - ]); - - const running = aggStatus.is_running || compStatus.is_running || autoStatus.is_running || leanojCurrentStatus.is_running; - setAnyWorkflowRunning(running); - } catch (error) { - console.error('Failed to check workflow status:', error); + const results = await Promise.allSettled([ + api.get('/api/aggregator/status'), + api.get('/api/compiler/status'), + autonomousAPI.getStatus(), + leanojAPI.getStatus(), + ]); + const fulfilled = results.filter((result) => result.status === 'fulfilled'); + const hasRunning = fulfilled.some((result) => Boolean(result.value?.is_running)); + if (hasRunning) { + setAnyWorkflowRunning(true); + } else if (fulfilled.length === results.length) { + setAnyWorkflowRunning(false); + } + for (const result of results) { + if (result.status === 'rejected') { + console.warn('Partial workflow status reconciliation failure:', result.reason); + } } }; @@ -3726,21 +3968,40 @@ function App() { return ( <div className={`app ${workflowPanelCollapsed ? 'workflow-panel-collapsed' : 'workflow-panel-expanded'}`}> - {/* Banner Section */} + {/* Product Brand Header */} <div className={`app-banner ${shimmerAccentsEnabled ? '' : 'no-shimmer'}`}> <div className="banner-content"> - <h1 className="banner-title"> - <span className="banner-moto" aria-label="M.O.T.O.">M.O.T.O.</span> - <span className="banner-subtitle">Autonomous ASI</span> - </h1> - <p className="banner-company">By Intrafere Research Group</p> - <p className="banner-variant">A Prototype Artificial Superintelligence - Novelty Seeking Autonomous S.T.E.M. Researcher For Automated Theorem Generation</p> - <p - className={`banner-mode-subtitle ${appMode === 'manual' || appMode === 'leanoj' ? '' : 'banner-mode-subtitle--hidden'}`} - aria-hidden={appMode !== 'manual' && appMode !== 'leanoj'} - > - {appMode === 'manual' ? 'MANUAL S.T.E.M. WRITER' : 'Proof Solver Mode'} - </p> + <div className="banner-brand"> + <a + className="banner-logo-link" + href="https://intrafere.com" + target="_blank" + rel="noreferrer" + aria-label="Visit Intrafere" + > + <img + className="banner-logo" + src={intrafereLogoMark} + alt="" + aria-hidden="true" + /> + </a> + <div className="banner-identity"> + <h1 className="banner-title"> + <span className="banner-moto" aria-label="MOTO">MOTO</span> + <span className="banner-subtitle">Autonomous ASI</span> + </h1> + <p className="banner-company">By Intrafere</p> + </div> + </div> + <div className="banner-positioning"> + <p + className={`banner-mode-subtitle ${appMode === 'manual' || appMode === 'leanoj' ? '' : 'banner-mode-subtitle--hidden'}`} + aria-hidden={appMode !== 'manual' && appMode !== 'leanoj'} + > + {appMode === 'manual' ? 'Manual S.T.E.M. Writer' : 'Proof Solver Mode'} + </p> + </div> </div> </div> @@ -3909,6 +4170,7 @@ function App() { status={autonomousStatus} activity={autonomousActivity} onStart={handleAutonomousStart} + onStartingChange={setAutonomousStarting} onStop={handleAutonomousStop} onClear={handleAutonomousClear} config={autonomousConfig} @@ -3963,7 +4225,7 @@ function App() { {activeTab === 'auto-completed-works' && ( <div className="completed-works-library"> <div className="completed-works-header"> - <h2 className="completed-works-title">Your Completed Works Library</h2> + <h2 className="completed-works-title">Completed Works Library</h2> <p className="completed-works-subtitle"> Browse all research outputs across every session — papers, final answers, and verified proofs. </p> @@ -4001,7 +4263,11 @@ function App() { <FinalAnswerLibrary capabilities={capabilities} /> )} {completedWorksSubTab === 'proof-library' && ( - <ProofLibrary /> + <ProofLibrary + selectedProofId={selectedHistoryProof?.corpus === 'moto' ? selectedHistoryProof.proofId : null} + selectedSessionId={selectedHistoryProof?.corpus === 'moto' ? selectedHistoryProof.sessionId : ''} + selectedRunId={selectedHistoryProof?.corpus === 'moto' ? selectedHistoryProof.runId : ''} + /> )} </div> </div> @@ -4051,6 +4317,9 @@ function App() { <LeanOJProofLibrary api={leanojAPI} refreshToken={leanojProofRefreshToken} + selectedProofId={selectedHistoryProof?.corpus === 'leanoj' ? selectedHistoryProof.proofId : null} + selectedSessionId={selectedHistoryProof?.corpus === 'leanoj' ? selectedHistoryProof.sessionId : ''} + selectedRunId={selectedHistoryProof?.corpus === 'leanoj' ? selectedHistoryProof.runId : ''} /> )} {activeTab === 'leanoj-logs' && ( @@ -4103,70 +4372,81 @@ function App() { {activeTab === 'compiler-logs' && <CompilerLogs />} {activeTab === 'compiler-live-paper' && <LivePaper capabilities={capabilities} />} {activeTab === 'compiler-proofs' && ( - <> - <MathematicalProofs - api={autonomousAPI} - refreshToken={proofRefreshToken} - selectedProofId={selectedProofScope === 'manual' ? selectedProofId : null} - proofScope="manual" - title="Manual Mathematical Proofs" - description="Active-run Lean 4 proofs generated from the manual Aggregator and manual Compiler. Clearing the manual run archives these proofs and removes them from future prompt context." - defaultSourceType="paper" - /> - <ProofLibrary - proofScope="manual" - title="Manual Proof Run History" - description="Archived manual proof runs. These proofs are available for review and download only; they are not injected into new manual runs." - /> - </> + <MathematicalProofs + api={autonomousAPI} + refreshToken={proofRefreshToken} + selectedProofId={selectedProofScope === 'manual' ? selectedProofId : null} + proofScope="manual" + title="Manual Mathematical Proofs" + description="Active-run Lean 4 proofs generated from the manual Aggregator and manual Compiler. Clearing the manual run archives these proofs and removes them from future prompt context." + defaultSourceType="paper" + /> + )} + {activeTab === 'compiler-proof-history' && ( + <ProofLibrary + proofScope="manual" + title="Manual Proof Run History" + description="Archived manual proof runs. These proofs are available for review and download only; they are not injected into new manual runs." + selectedProofId={selectedHistoryProof?.corpus === 'manual' ? selectedHistoryProof.proofId : null} + selectedSessionId={selectedHistoryProof?.corpus === 'manual' ? selectedHistoryProof.sessionId : ''} + selectedRunId={selectedHistoryProof?.corpus === 'manual' ? selectedHistoryProof.runId : ''} + /> )} </div> </div> {/* Autonomous Settings - Rendered OUTSIDE tab-content to allow full-width sidebar layout */} {activeTab === 'auto-settings' && ( - <AutonomousResearchSettings - config={autonomousConfig} - onConfigChange={setAutonomousConfig} - models={models} - capabilities={capabilities} - connectivityStatus={connectivityStatus} - credentialStatusRefreshToken={credentialStatusRefreshToken} - isRunning={autonomousRunning} - developerModeEnabled={developerModeEnabled} - /> + <div className="settings-overlay-safe-area"> + <AutonomousResearchSettings + config={autonomousConfig} + onConfigChange={setAutonomousConfig} + models={models} + capabilities={capabilities} + connectivityStatus={connectivityStatus} + credentialStatusRefreshToken={credentialStatusRefreshToken} + isRunning={autonomousRunning || autonomousStarting || autonomousStopping} + developerModeEnabled={developerModeEnabled} + /> + </div> )} {activeTab === 'leanoj-settings' && ( - <LeanOJSettings - settings={leanojSettings} - onSettingsChange={setLeanojSettings} - capabilities={capabilities} - connectivityStatus={connectivityStatus} - credentialStatusRefreshToken={credentialStatusRefreshToken} - isRunning={leanojRunning} - developerModeEnabled={developerModeEnabled} - /> + <div className="settings-overlay-safe-area"> + <LeanOJSettings + settings={leanojSettings} + onSettingsChange={setLeanojSettings} + capabilities={capabilities} + connectivityStatus={connectivityStatus} + credentialStatusRefreshToken={credentialStatusRefreshToken} + isRunning={leanojRunning} + developerModeEnabled={developerModeEnabled} + /> + </div> )} {activeTab === 'aggregator-settings' && ( - <AggregatorSettings - config={config} - setConfig={setConfig} - capabilities={capabilities} - connectivityStatus={connectivityStatus} - credentialStatusRefreshToken={credentialStatusRefreshToken} - developerModeEnabled={developerModeEnabled} - /> + <div className="settings-overlay-safe-area"> + <AggregatorSettings + config={config} + setConfig={setConfig} + capabilities={capabilities} + connectivityStatus={connectivityStatus} + credentialStatusRefreshToken={credentialStatusRefreshToken} + developerModeEnabled={developerModeEnabled} + /> + </div> )} {activeTab === 'compiler-settings' && ( - <CompilerSettings - capabilities={capabilities} - connectivityStatus={connectivityStatus} - credentialStatusRefreshToken={credentialStatusRefreshToken} - developerModeEnabled={developerModeEnabled} - /> + <div className="settings-overlay-safe-area"> + <CompilerSettings + capabilities={capabilities} + connectivityStatus={connectivityStatus} + credentialStatusRefreshToken={credentialStatusRefreshToken} + developerModeEnabled={developerModeEnabled} + /> + </div> )} {/* WorkflowPanel is ETERNAL - always visible for boost controls */} @@ -4175,9 +4455,45 @@ function App() { <WorkflowPanel isRunning={anyWorkflowRunning} onOpenBoostSettings={() => setShowBoostModal(true)} + onOpenAssistantProof={handleOpenAssistantProof} + onOpenSolutionPath={(snapshot) => { + solutionPathFenceRef.current = snapshot; + setSolutionPathSnapshot(snapshot); + }} + onSolutionPathSnapshotChange={(snapshot) => { + solutionPathFenceRef.current = snapshot; + setSolutionPathSnapshot((current) => ( + current && current.run_id === snapshot?.run_id ? snapshot : current + )); + }} collapsed={workflowPanelCollapsed} onCollapseChange={setWorkflowPanelCollapsed} /> + {solutionPathSnapshot && ( + <SolutionPathModal + snapshot={solutionPathSnapshot} + onClose={() => setSolutionPathSnapshot(null)} + onSnapshotChange={(snapshot) => { + solutionPathFenceRef.current = snapshot; + setSolutionPathSnapshot((current) => ( + !current || current.run_id === snapshot?.run_id ? snapshot : current + )); + }} + onOpenSettings={(mode) => { + setSolutionPathSnapshot(null); + if (mode === 'leanoj') { + setAppMode('leanoj'); + setLeanojActiveTab('leanoj-settings'); + } else if (mode === 'autonomous') { + setAppMode('autonomous'); + setAutonomousActiveTab('auto-settings'); + } else { + setAppMode('manual'); + setManualActiveTab(mode === 'compiler' ? 'compiler-settings' : 'aggregator-settings'); + } + }} + /> + )} {/* Disclaimer Modal - Shows on every app load */} {showDisclaimer && ( diff --git a/frontend/src/assets/brand/intrafere-logo-no-text.png b/frontend/src/assets/brand/intrafere-logo-no-text.png new file mode 100644 index 0000000..c1625a0 Binary files /dev/null and b/frontend/src/assets/brand/intrafere-logo-no-text.png differ diff --git a/frontend/src/assets/brand/intrafere-logo.png b/frontend/src/assets/brand/intrafere-logo.png new file mode 100644 index 0000000..fa0bcfb Binary files /dev/null and b/frontend/src/assets/brand/intrafere-logo.png differ diff --git a/frontend/src/components/HighlightedModelsSidebar.jsx b/frontend/src/components/HighlightedModelsSidebar.jsx index ea701fb..0519260 100644 --- a/frontend/src/components/HighlightedModelsSidebar.jsx +++ b/frontend/src/components/HighlightedModelsSidebar.jsx @@ -83,7 +83,7 @@ export default function HighlightedModelsSidebar() { <OauthTag stacked /> <div className="flex-row-center"> <ProofStrengthBadge variant="leaderboard" className="ps-badge-anchor--model-only" /> - <div className="model-item-name">ChatGPT 5.5</div> + <div className="model-item-name">GPT 5.6</div> <div className="ranking-badge ranking-badge--silver">🥈 SILVER</div> </div> <div className="model-item-badge">Powerful and affordable OAuth</div> @@ -92,7 +92,7 @@ export default function HighlightedModelsSidebar() { <div className="model-item model-item--ranked model-item--bronze model-item--oa"> <OauthTag /> <div className="flex-row-center"> - <div className="model-item-name">Grok 4.3</div> + <div className="model-item-name">Grok 4.5</div> <div className="ranking-badge ranking-badge--bronze">🥉 BRONZE</div> </div> <div className="model-item-badge">Powerful and affordable OAuth</div> @@ -105,24 +105,50 @@ export default function HighlightedModelsSidebar() { </div> <div className="model-item"> - <div className="model-item-name">Amazon Nova Pro/Premier</div> + <div className="model-item-name">Amazon's Nova</div> <div className="model-item-badge">Highly knowledgeable</div> </div> <div className="model-item"> <ProofStrengthBadge variant="leaderboard" className="ps-badge-anchor--model-only" /> - <div className="model-item-name">Claude Opus/Sonnet/Haiku</div> + <div className="model-item-name">Anthropic's Claude</div> + <div className="model-item-badge">Highly knowledgeable</div> + </div> + + <div className="model-item"> + <div className="model-item-name">Cohere's Command A</div> + <div className="model-item-badge">Highly knowledgeable</div> + </div> + + <div className="model-item"> + <div className="model-item-name">Mistral AI's Mistral Large</div> + <div className="model-item-badge">Highly knowledgeable</div> + </div> + + <div className="model-item model-item--os"> + <OsTag /> + <div className="model-item-name">Meta's Llama</div> + <div className="model-item-badge">Highly knowledgeable</div> + </div> + + <div className="model-item"> + <div className="model-item-name">Moonshot AI's Kimi</div> + <div className="model-item-badge">Highly knowledgeable</div> + </div> + + <div className="model-item"> + <div className="model-item-name">AI21's Jamba</div> <div className="model-item-badge">Highly knowledgeable</div> </div> <div className="model-item model-item--os"> <OsTag /> - <div className="model-item-name">DeepSeek</div> + <div className="model-item-name">DeepSeek's DeepSeek</div> <div className="model-item-badge">Highly knowledgeable</div> </div> <div className="model-item"> - <div className="model-item-name">Gemini Pro</div> + <div className="model-item-name">Google's Gemini Pro</div> <div className="model-item-badge">Highly knowledgeable</div> </div> @@ -134,13 +160,13 @@ export default function HighlightedModelsSidebar() { <div className="model-item model-item--os"> <OsTag /> - <div className="model-item-name">GLM</div> + <div className="model-item-name">Z.AI's GLM</div> <div className="model-item-badge">Highly knowledgeable</div> </div> <div className="model-item model-item--os"> <OsTag /> - <div className="model-item-name">GLM Turbo</div> + <div className="model-item-name">Z.AI's GLM Turbo</div> <div className="model-item-badge">Fast validator</div> </div> @@ -152,21 +178,21 @@ export default function HighlightedModelsSidebar() { <div className="model-item model-item--oa"> <OauthTag /> - <div className="model-item-name">Grok</div> + <div className="model-item-name">xAI's Grok</div> <div className="model-item-badge">Highly knowledgeable</div> </div> <div className="model-item model-item--oa"> <OauthTag stacked /> <ProofStrengthBadge variant="leaderboard" className="ps-badge-anchor--model-only" /> - <div className="model-item-name">ChatGPT</div> + <div className="model-item-name">OpenAI's ChatGPT</div> <div className="model-item-badge">Highly knowledgeable</div> </div> <div className="model-item model-item--oa"> <OauthTag stacked /> <ProofStrengthBadge variant="leaderboard" className="ps-badge-anchor--model-only" /> - <div className="model-item-name">Sakana Fugu</div> + <div className="model-item-name">Sakana AI's Fugu</div> <div className="model-item-badge">Highly knowledgeable model fusion</div> </div> @@ -177,13 +203,13 @@ export default function HighlightedModelsSidebar() { <div className="model-item model-item--os"> <OsTag /> - <div className="model-item-name">Nemotron Super</div> + <div className="model-item-name">NVIDIA's Nemotron Super</div> <div className="model-item-badge">Balanced knowledge and speed</div> </div> <div className="model-item model-item--os"> <OsTag /> - <div className="model-item-name">Nous Hermes</div> + <div className="model-item-name">Nous Research's Hermes</div> <div className="model-item-badge">Highly knowledgeable</div> </div> @@ -199,19 +225,19 @@ export default function HighlightedModelsSidebar() { </div> <div className="model-item"> - <div className="model-item-name">MiniMax</div> + <div className="model-item-name">MiniMax's MiniMax</div> <div className="model-item-badge">Highly knowledgeable</div> </div> <div className="model-item model-item--os"> <OsTag /> - <div className="model-item-name">Qwen Coder</div> + <div className="model-item-name">Alibaba's Qwen Coder</div> <div className="model-item-badge">Computer science</div> </div> <div className="model-item model-item--os"> <OsTag /> - <div className="model-item-name">Qwen</div> + <div className="model-item-name">Alibaba's Qwen</div> <div className="model-item-badge">Highly knowledgeable</div> </div> </div> diff --git a/frontend/src/components/LatexRenderer.jsx b/frontend/src/components/LatexRenderer.jsx index a348474..89ec43e 100644 --- a/frontend/src/components/LatexRenderer.jsx +++ b/frontend/src/components/LatexRenderer.jsx @@ -1029,14 +1029,6 @@ const LatexRenderer = ({ ); const [largeDocWarningDismissed, setLargeDocWarningDismissed] = useState(false); - // Auto-switch to raw when content grows past threshold (for live/growing documents). - // Only fires if the user has not explicitly opted into rendered mode. - useEffect(() => { - if (isLargeDoc && !largeDocWarningDismissed) { - setInternalViewMode('raw'); - } - }, [isLargeDoc, largeDocWarningDismissed]); - const debouncedContent = useDebouncedValue(content, 1500); const viewMode = showLatex !== undefined diff --git a/frontend/src/components/OpenRouterFreeModelsControl.jsx b/frontend/src/components/OpenRouterFreeModelsControl.jsx new file mode 100644 index 0000000..8c12755 --- /dev/null +++ b/frontend/src/components/OpenRouterFreeModelsControl.jsx @@ -0,0 +1,37 @@ +import React from 'react'; +import HelpTooltip from './HelpTooltip'; + +const FREE_MODELS_HELP_TEXT = + 'If you have more than $10 in your OpenRouter account, OpenRouter provides 1,000 free API calls per day that you can use with any program. These free models are useful as validators or supplemental brainstorm roles. They are typically not as knowledgeable as state-of-the-art models. Checking "Only show free OpenRouter models" filters the OpenRouter model lists to show only free models.'; + +export default function OpenRouterFreeModelsControl({ + checked, + disabled = false, + onChange, +}) { + return ( + <span className="openrouter-free-models-control"> + <label className="openrouter-free-models-control__checkbox"> + <input + type="checkbox" + checked={checked} + disabled={disabled} + onChange={(event) => onChange(event.target.checked)} + /> + Only show free OpenRouter models + </label> + <span className="openrouter-free-models-control__help"> + <span className="openrouter-free-models-control__help-label">Run MOTO for free</span> + <HelpTooltip + label="Run MOTO for free" + buttonClassName="openrouter-free-models-control__help-button" + popupClassName="openrouter-free-models-control__tooltip" + useFixedPosition + fixedPlacement="side-right" + > + {FREE_MODELS_HELP_TEXT} + </HelpTooltip> + </span> + </span> + ); +} diff --git a/frontend/src/components/SolutionPathModal.css b/frontend/src/components/SolutionPathModal.css new file mode 100644 index 0000000..3ad5f00 --- /dev/null +++ b/frontend/src/components/SolutionPathModal.css @@ -0,0 +1,42 @@ +.solution-path-modal__backdrop { + position: fixed; inset: 0; z-index: 1000000; display: grid; place-items: center; + padding: 24px; background: rgba(5, 8, 18, .76); backdrop-filter: blur(5px); +} +.solution-path-modal { + width: min(720px, 100%); max-height: min(82vh, 800px); overflow: auto; + border: 1px solid rgba(139, 116, 255, .4); border-radius: 18px; padding: 22px; + color: #f5f3ff; background: #121426; box-shadow: 0 24px 80px rgba(0, 0, 0, .5); +} +.solution-path-modal__header { display: flex; justify-content: space-between; gap: 16px; align-items: start; } +.solution-path-modal__header h2 { margin: 4px 0 0; } +.solution-path-modal__header button { border: 0; background: transparent; color: inherit; font-size: 28px; cursor: pointer; } +.solution-path-modal__eyebrow { color: #a99bff; font-size: 12px; text-transform: uppercase; letter-spacing: .08em; } +.solution-path-modal__meta { display: flex; flex-wrap: wrap; gap: 8px; margin: 16px 0; } +.solution-path-modal__meta span, .solution-path-modal__status { + padding: 4px 8px; border-radius: 999px; background: rgba(139, 116, 255, .13); font-size: 12px; +} +.solution-path-modal__steps { list-style: none; margin: 0; padding: 0; } +.solution-path-modal__step { display: grid; grid-template-columns: 18px 1fr; gap: 12px; padding: 12px 0; } +.solution-path-modal__marker { width: 12px; height: 12px; margin-top: 5px; border-radius: 50%; background: #676b80; } +.solution-path-modal__step.status-active .solution-path-modal__marker { background: #a99bff; box-shadow: 0 0 12px #806cff; } +.solution-path-modal__step.status-complete .solution-path-modal__marker { background: #5bd89b; } +.solution-path-modal__step.status-blocked .solution-path-modal__marker { background: #ef6f7d; } +.solution-path-modal__step-title { font-weight: 700; } +.solution-path-modal__step p { margin: 5px 0 8px; color: #c8c8d8; line-height: 1.45; } +.solution-path-modal__empty { color: #c8c8d8; } +.solution-path-modal__repair-count { color: #ffd7a3; background: rgba(239, 151, 60, .18) !important; } +.solution-path-modal__repair { + margin: 16px 0; padding: 14px; border: 1px solid rgba(239, 151, 60, .55); + border-radius: 12px; background: rgba(239, 151, 60, .1); +} +.solution-path-modal__repair p { margin: 8px 0; color: #e4deef; line-height: 1.45; } +.solution-path-modal__repair h3 { margin: 0; color: #ffd7a3; } +.solution-path-modal__repair-item { padding-top: 10px; border-top: 1px solid rgba(239, 151, 60, .25); } +.solution-path-modal__repair-reason { font-size: 13px; color: #ffd7a3 !important; } +.solution-path-modal__repair-error, .solution-path-modal__retry-error { color: #ff9aa6 !important; } +.solution-path-modal__repair-actions { display: flex; flex-wrap: wrap; gap: 10px; margin-top: 12px; } +.solution-path-modal__repair-actions button { + padding: 8px 12px; border: 1px solid rgba(169, 155, 255, .55); border-radius: 8px; + color: #f5f3ff; background: rgba(139, 116, 255, .18); cursor: pointer; +} +.solution-path-modal__repair-actions button:disabled { cursor: wait; opacity: .65; } diff --git a/frontend/src/components/SolutionPathModal.jsx b/frontend/src/components/SolutionPathModal.jsx new file mode 100644 index 0000000..1de9408 --- /dev/null +++ b/frontend/src/components/SolutionPathModal.jsx @@ -0,0 +1,241 @@ +import React, { useEffect, useRef, useState } from 'react'; +import { createPortal } from 'react-dom'; +import { workflowAPI } from '../services/api'; +import { websocket } from '../services/websocket'; +import { + getSolutionPathSettingsMode, + isSolutionPathSnapshotAtLeast, + solutionPathEventMatches, +} from '../utils/solutionPathPresentation'; +import './SolutionPathModal.css'; + +export default function SolutionPathModal({ snapshot, onClose, onSnapshotChange, onOpenSettings }) { + const dialogRef = useRef(null); + const closeButtonRef = useRef(null); + const snapshotRef = useRef(snapshot); + const retryButtonRefs = useRef(new Map()); + const [retryingProposalId, setRetryingProposalId] = useState(''); + const [retryError, setRetryError] = useState(''); + const [retryStatus, setRetryStatus] = useState(''); + + useEffect(() => { + snapshotRef.current = snapshot; + }, [snapshot]); + + useEffect(() => { + let requestSequence = 0; + const refresh = async (event = {}) => { + const current = snapshotRef.current; + if (!solutionPathEventMatches(event, current, current?.mode || '')) return; + const sequence = ++requestSequence; + try { + const nextSnapshot = await workflowAPI.getSolutionPath(); + if ( + sequence === requestSequence + && nextSnapshot?.run_id === current?.run_id + && isSolutionPathSnapshotAtLeast(nextSnapshot, snapshotRef.current) + ) { + snapshotRef.current = nextSnapshot; + onSnapshotChange?.(nextSnapshot); + } + } catch { + // The next workflow event or explicit retry will refresh the modal. + } + }; + const events = [ + 'solution_path_proposal_queued', + 'solution_path_proposal_reviewing', + 'solution_path_updated', + 'solution_path_proposal_rejected', + 'solution_path_proposal_retry_queued', + 'solution_path_proposal_user_repair_required', + 'solution_path_proposal_resumed', + ]; + events.forEach((eventName) => websocket.on(eventName, refresh)); + return () => { + requestSequence += 1; + events.forEach((eventName) => websocket.off(eventName, refresh)); + }; + }, [onSnapshotChange]); + useEffect(() => { + const previouslyFocused = document.activeElement; + closeButtonRef.current?.focus(); + const handleKeyDown = (event) => { + if (event.key === 'Escape') onClose(); + if (event.key !== 'Tab' || !dialogRef.current) return; + const focusable = dialogRef.current.querySelectorAll( + 'button, [href], input, select, textarea, [tabindex]:not([tabindex="-1"])' + ); + if (!focusable.length) { + event.preventDefault(); + dialogRef.current.focus(); + return; + } + const first = focusable[0]; + const last = focusable[focusable.length - 1]; + if (event.shiftKey && document.activeElement === first) { + event.preventDefault(); + last.focus(); + } else if (!event.shiftKey && document.activeElement === last) { + event.preventDefault(); + first.focus(); + } + }; + window.addEventListener('keydown', handleKeyDown); + return () => { + window.removeEventListener('keydown', handleKeyDown); + previouslyFocused?.focus?.(); + }; + }, [onClose]); + + if (!snapshot || typeof document === 'undefined') return null; + const steps = Array.isArray(snapshot.steps) ? snapshot.steps : []; + const repairs = Array.isArray(snapshot.repairs) ? snapshot.repairs : []; + + const retryRepair = async (repair) => { + setRetryError(''); + setRetryStatus(''); + setRetryingProposalId(repair.proposal_id); + const retryFence = { + runId: snapshot.run_id, + lifecycleGeneration: repair.lifecycle_generation || snapshot.lifecycle_generation, + proposalId: repair.proposal_id, + }; + try { + await workflowAPI.resumeSolutionPathProposal({ + runId: retryFence.runId, + proposalId: retryFence.proposalId, + lifecycleGeneration: retryFence.lifecycleGeneration, + }); + const nextSnapshot = await workflowAPI.getSolutionPath(); + if ( + nextSnapshot?.run_id !== retryFence.runId + || Number(nextSnapshot?.lifecycle_generation) !== Number(retryFence.lifecycleGeneration) + ) { + throw new Error('The active solution-path run changed. Reopen the current route before retrying.'); + } + if (isSolutionPathSnapshotAtLeast(nextSnapshot, snapshotRef.current)) { + snapshotRef.current = nextSnapshot; + onSnapshotChange?.(nextSnapshot); + } + setRetryStatus('Review retry queued. The route will refresh as the review progresses.'); + } catch (error) { + setRetryError(error.message || 'The solution-path update could not be retried.'); + } finally { + setRetryingProposalId(''); + requestAnimationFrame(() => retryButtonRefs.current.get(repair.proposal_id)?.focus()); + } + }; + + return createPortal( + <div className="solution-path-modal__backdrop" role="presentation" onMouseDown={onClose}> + <section + ref={dialogRef} + className="solution-path-modal" + role="dialog" + aria-modal="true" + aria-labelledby="solution-path-title" + aria-describedby="solution-path-guidance" + tabIndex={-1} + onMouseDown={(event) => event.stopPropagation()} + > + <header className="solution-path-modal__header"> + <div> + <div className="solution-path-modal__eyebrow">{snapshot.mode || 'Workflow'} · Solution Path</div> + <h2 id="solution-path-title">Current solution path</h2> + </div> + <button ref={closeButtonRef} type="button" onClick={onClose} aria-label="Close solution path">×</button> + </header> + <div className="solution-path-modal__meta"> + {snapshot.revision && <span>Revision {snapshot.revision}</span>} + {snapshot.run_id && <span>Run {snapshot.run_id}</span>} + {snapshot.lifecycle_generation != null && <span>Generation {snapshot.lifecycle_generation}</span>} + {snapshot.ownership && <span>{snapshot.ownership === 'resumable' ? 'Resumable run' : 'Active run'}</span>} + <span>{snapshot.ordering === 'unordered' ? 'Flexible order' : 'Ordered route'}</span> + {snapshot.pending_proposals > 0 && <span>{snapshot.pending_proposals} update(s) under review</span>} + </div> + <p id="solution-path-guidance" className="solution-path-modal__guidance"> + This is a distillation attempt at the best currently known path from the available context. + It may be wrong or incomplete, is optional guidance, and may be deviated from whenever a + better route serves the user prompt or current subgoal. + </p> + {repairs.length > 0 && ( + <section className="solution-path-modal__repair" aria-labelledby="solution-path-repair-title"> + <h3 id="solution-path-repair-title">Settings repair required</h3> + <p> + Update the Main Submitter 1 provider, credentials, or context settings, then retry the + blocked review. The workflow remains resumable. + </p> + {repairs.map((repair) => ( + <div key={repair.proposal_id} className="solution-path-modal__repair-item"> + <strong>{String(repair.reason || 'reviewer failure').replaceAll('_', ' ')}</strong> + {repair.detail && <p>{repair.detail}</p>} + <div className="solution-path-modal__repair-actions"> + <button type="button" onClick={() => onOpenSettings?.(getSolutionPathSettingsMode(snapshot))}> + Open Main Submitter 1 settings + </button> + <button + ref={(node) => { + if (node) retryButtonRefs.current.set(repair.proposal_id, node); + else retryButtonRefs.current.delete(repair.proposal_id); + }} + type="button" + onClick={() => retryRepair(repair)} + disabled={Boolean(retryingProposalId)} + > + {retryingProposalId === repair.proposal_id ? 'Retrying…' : 'Retry review'} + </button> + </div> + </div> + ))} + {retryError && <p className="solution-path-modal__repair-error" role="alert">{retryError}</p>} + {retryStatus && <p className="solution-path-modal__repair-status" role="status" aria-live="polite">{retryStatus}</p>} + </section> + )} + {snapshot.main_route && ( + <section className="solution-path-modal__main-route" aria-label="Main route to solution"> + <strong>Main route to solution</strong> + <p>{snapshot.main_route}</p> + </section> + )} + {!snapshot.enabled || steps.length === 0 ? ( + <div className="solution-path-modal__empty"> + <strong>No approved route yet</strong> + <p>{snapshot.message || 'No solution path is available yet.'}</p> + {snapshot.acceptance_count >= 5 && ( + <small>{snapshot.acceptance_count} accepted brainstorm ideas</small> + )} + </div> + ) : ( + React.createElement( + snapshot.ordering === 'unordered' ? 'ul' : 'ol', + { + className: `solution-path-modal__steps ${snapshot.ordering === 'unordered' ? 'unordered' : ''}`, + 'aria-label': snapshot.ordering === 'unordered' + ? 'Flexible solution route steps' + : 'Ordered solution route steps', + }, + steps.map((step) => ( + <li + key={step.step_id} + className={`solution-path-modal__step status-${step.status}`} + aria-label={`${step.title}: ${step.status}`} + > + <span className="solution-path-modal__marker" aria-hidden="true" /> + <div> + <div className="solution-path-modal__step-title">{step.title}</div> + {step.description && <p>{step.description}</p>} + <span className="solution-path-modal__status"> + <span aria-hidden="true">{step.status === 'complete' ? '✓ ' : ''}</span> + {step.status} + </span> + </div> + </li> + )) + ) + )} + </section> + </div>, + document.body + ); +} diff --git a/frontend/src/components/SolutionPathModal.test.jsx b/frontend/src/components/SolutionPathModal.test.jsx new file mode 100644 index 0000000..07dc0fa --- /dev/null +++ b/frontend/src/components/SolutionPathModal.test.jsx @@ -0,0 +1,105 @@ +import React from 'react'; +import { render, screen, waitFor } from '@testing-library/react'; +import userEvent from '@testing-library/user-event'; +import SolutionPathModal from './SolutionPathModal'; +import { workflowAPI } from '../services/api'; + +vi.mock('../services/websocket', () => ({ + websocket: { on: vi.fn(), off: vi.fn() }, +})); + +vi.mock('../services/api', () => ({ + workflowAPI: { + getSolutionPath: vi.fn(), + resumeSolutionPathProposal: vi.fn(), + }, +})); + +const repairSnapshot = { + enabled: false, + ownership: 'active', + mode: 'compiler', + run_id: 'manual-run', + lifecycle_generation: 4, + acceptance_count: 7, + revision: null, + steps: [], + repairs: [{ + proposal_id: 'proposal-1', + reason: 'context_overflow', + detail: 'Increase the context window.', + lifecycle_generation: 4, + }], + message: 'Solution path tracking is loaded; no approved plan is available yet.', +}; + +beforeEach(() => { + vi.clearAllMocks(); +}); + +test('routes compiler repair to Aggregator Main Submitter settings and presents no-plan metadata', async () => { + const user = userEvent.setup(); + const onOpenSettings = vi.fn(); + render( + <SolutionPathModal + snapshot={repairSnapshot} + onClose={vi.fn()} + onSnapshotChange={vi.fn()} + onOpenSettings={onOpenSettings} + /> + ); + + expect(screen.getByText('No approved route yet')).toBeInTheDocument(); + expect(screen.getByText(/7 accepted brainstorm ideas/i)).toBeInTheDocument(); + expect(screen.getByText('Run manual-run')).toBeInTheDocument(); + expect(screen.getByText('Generation 4')).toBeInTheDocument(); + + await user.click(screen.getByRole('button', { name: /Open Main Submitter 1 settings/i })); + expect(onOpenSettings).toHaveBeenCalledWith('aggregator'); +}); + +test('refreshes a retry only within the same run and generation and announces success', async () => { + const user = userEvent.setup(); + const onSnapshotChange = vi.fn(); + const refreshed = { ...repairSnapshot, repairs: [], queued_proposals: 1 }; + workflowAPI.resumeSolutionPathProposal.mockResolvedValue({ success: true }); + workflowAPI.getSolutionPath.mockResolvedValue(refreshed); + + render( + <SolutionPathModal + snapshot={repairSnapshot} + onClose={vi.fn()} + onSnapshotChange={onSnapshotChange} + onOpenSettings={vi.fn()} + /> + ); + const retryButton = screen.getByRole('button', { name: 'Retry review' }); + await user.click(retryButton); + + await waitFor(() => expect(onSnapshotChange).toHaveBeenCalledWith(refreshed)); + expect(screen.getByRole('status')).toHaveTextContent(/retry queued/i); + await waitFor(() => expect(retryButton).toHaveFocus()); +}); + +test('rejects a retry refresh from a different lifecycle generation', async () => { + const user = userEvent.setup(); + const onSnapshotChange = vi.fn(); + workflowAPI.resumeSolutionPathProposal.mockResolvedValue({ success: true }); + workflowAPI.getSolutionPath.mockResolvedValue({ + ...repairSnapshot, + lifecycle_generation: 5, + }); + + render( + <SolutionPathModal + snapshot={repairSnapshot} + onClose={vi.fn()} + onSnapshotChange={onSnapshotChange} + onOpenSettings={vi.fn()} + /> + ); + await user.click(screen.getByRole('button', { name: 'Retry review' })); + + expect(await screen.findByRole('alert')).toHaveTextContent(/run changed/i); + expect(onSnapshotChange).not.toHaveBeenCalled(); +}); diff --git a/frontend/src/components/SyntheticLib4AccessModal.jsx b/frontend/src/components/SyntheticLib4AccessModal.jsx index e29e346..0d77f6e 100644 --- a/frontend/src/components/SyntheticLib4AccessModal.jsx +++ b/frontend/src/components/SyntheticLib4AccessModal.jsx @@ -166,7 +166,7 @@ export default function SyntheticLib4AccessModal({ <div style={sectionStyle}> <h3 style={sectionHeadingStyle}>Contribution And Access</h3> <p style={bodyTextStyle}> - Users will be able to select 30 novel proofs to contribute. The database is only for novel solutions that are not already known to other proof databases, and many accepted solutions are expected to be too new for models to have trained on yet. Your local system must first deem them novel, then the SyntheticLib4 vetting system will double-check the results and decide whether they are contribution-worthy. If accepted, the contribution grants one month of access. Users can continue to contribute to extend their subscription beyond their initial month. + Users will be able to select 20 novel proofs to contribute. The database is only for novel solutions that are not already known to other proof databases, and many accepted solutions are expected to be too new for models to have trained on yet. Your local system must first deem them novel, then the SyntheticLib4 vetting system will double-check the results and decide whether they are contribution-worthy. If accepted, the contribution grants one month of access. Users can continue to contribute to extend their subscription beyond their initial month. </p> <p style={{ ...bodyTextStyle, marginBottom: 0 }}> Users who do not want to share proof data for access will be able to pay a small monthly fee instead. That fee supports ongoing model review, pruning, infrastructure, and ecosystem maintenance. The only data shared with our system is proof data manually selected by users; all other data remains private and local to users' computers. diff --git a/frontend/src/components/SyntheticLib4AccessModal.test.jsx b/frontend/src/components/SyntheticLib4AccessModal.test.jsx index 4525be7..0dcd9d2 100644 --- a/frontend/src/components/SyntheticLib4AccessModal.test.jsx +++ b/frontend/src/components/SyntheticLib4AccessModal.test.jsx @@ -22,14 +22,14 @@ test('shows SyntheticLib4 coming-soon explainer copy', () => { 'href', 'https://x.com/IntrafereLLC' ); - expect(screen.getByText(/super-MOTO brainstorm database/i)).toBeInTheDocument(); - expect(screen.getByText(/value for their data with their consent/i)).toBeInTheDocument(); + expect(screen.getByText(/contribution-based Lean 4 proof ecosystem/i)).toBeInTheDocument(); + expect(screen.getByText(/unused proofs can now provide value back to you/i)).toBeInTheDocument(); }); test('explains contribution access and redistribution limits', () => { renderModal(); - expect(screen.getByText(/30 novel proofs/i)).toBeInTheDocument(); + expect(screen.getByText(/20 novel proofs/i)).toBeInTheDocument(); expect(screen.getByText(/one month of access/i)).toBeInTheDocument(); expect(screen.getByText(/pay a small monthly fee/i)).toBeInTheDocument(); expect(screen.getByText(/cite, use, and reference individual proofs/i)).toBeInTheDocument(); diff --git a/frontend/src/components/TextFileUploader.css b/frontend/src/components/TextFileUploader.css index 0d19392..bffddab 100644 --- a/frontend/src/components/TextFileUploader.css +++ b/frontend/src/components/TextFileUploader.css @@ -124,6 +124,57 @@ outline-offset: 2px; } +.attached-file-list { + display: flex; + flex-wrap: wrap; + gap: 0.5rem; + margin-top: 0.75rem; +} + +.attached-file-chip { + display: inline-flex; + align-items: center; + gap: 0.35rem; + max-width: 100%; + padding: 0.35rem 0.5rem; + border: 1px solid var(--accent-green, #4CAF50); + border-radius: 999px; + background: rgba(76, 175, 80, 0.12); + color: var(--accent-green, #4CAF50); + font-size: 0.85rem; +} + +.attached-file-name { + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; +} + +.attached-file-remove { + display: inline-flex; + align-items: center; + justify-content: center; + width: 18px; + height: 18px; + padding: 0; + border: none; + border-radius: 50%; + background: transparent; + color: inherit; + cursor: pointer; + font-size: 1rem; + line-height: 1; +} + +.attached-file-remove:hover:not(:disabled) { + background: rgba(76, 175, 80, 0.2); +} + +.attached-file-remove:disabled { + cursor: not-allowed; + opacity: 0.6; +} + /* Mobile Responsiveness - Matches existing breakpoint */ @media (max-width: 768px) { .text-upload-btn { diff --git a/frontend/src/components/TextFileUploader.jsx b/frontend/src/components/TextFileUploader.jsx index 5a450d2..b3ea392 100644 --- a/frontend/src/components/TextFileUploader.jsx +++ b/frontend/src/components/TextFileUploader.jsx @@ -2,8 +2,8 @@ import React, { useState, useRef } from 'react'; import './TextFileUploader.css'; /** - * TextFileUploader - Single .txt file upload component for prompt enhancement - * Reads file content and merges it into the prompt textarea (chat-style AI interface pattern) + * TextFileUploader - .txt or .lean file upload component for prompt context + * Reads file contents and passes them to the parent for prompt/context handling. */ function TextFileUploader({ onFileLoaded, @@ -24,8 +24,8 @@ function TextFileUploader({ }; const handleFileChange = async (event) => { - const file = event.target.files?.[0]; - if (!file) return; + const files = Array.from(event.target.files || []); + if (files.length === 0) return; // Reset file input to allow re-uploading the same file event.target.value = ''; @@ -33,31 +33,35 @@ function TextFileUploader({ // Clear previous status setStatus(null); - // Validate file type - if (!file.name.toLowerCase().endsWith('.txt')) { - setStatus({ - type: 'error', - message: 'Please select a .txt file' - }); - autoHideStatus(); - return; - } + for (const file of files) { + const lowerFileName = file.name.toLowerCase(); + const isAllowedFile = lowerFileName.endsWith('.txt') || lowerFileName.endsWith('.lean'); - // Validate file size - const fileSizeMB = file.size / (1024 * 1024); - if (fileSizeMB > maxSizeMB) { - setStatus({ - type: 'error', - message: `File exceeds ${maxSizeMB}MB limit (actual: ${fileSizeMB.toFixed(2)}MB)` - }); - autoHideStatus(); - return; + if (!isAllowedFile) { + setStatus({ + type: 'error', + message: 'Please select only .txt or .lean files' + }); + autoHideStatus(); + return; + } + + const fileSizeMB = file.size / (1024 * 1024); + if (fileSizeMB > maxSizeMB) { + setStatus({ + type: 'error', + message: `${file.name} exceeds ${maxSizeMB}MB limit (actual: ${fileSizeMB.toFixed(2)}MB)` + }); + autoHideStatus(); + return; + } } // Show confirmation if prompt already has content if (confirmIfNotEmpty && existingPromptLength > 100) { + const fileText = files.length === 1 ? 'the file' : `${files.length} files`; const confirmed = window.confirm( - 'Prompt already contains text. This will ADD the file content to the end. Continue?' + `Prompt already contains text. This will add ${fileText} as labeled context blocks at the end. Continue?` ); if (!confirmed) { return; @@ -67,27 +71,32 @@ function TextFileUploader({ // Read file setIsLoading(true); try { - const content = await readFileContent(file); - - // Strip BOM if present (UTF-8 BOM: 0xEF 0xBB 0xBF) - const cleanContent = content.charCodeAt(0) === 0xFEFF ? content.slice(1) : content; + const loadedFiles = []; - // Handle empty or whitespace-only files - if (!cleanContent.trim()) { - setStatus({ - type: 'error', - message: 'File is empty or contains only whitespace' - }); - autoHideStatus(); - return; + for (const file of files) { + const content = await readFileContent(file); + + // Strip BOM if present (UTF-8 BOM: 0xEF 0xBB 0xBF) + const cleanContent = content.charCodeAt(0) === 0xFEFF ? content.slice(1) : content; + + if (!cleanContent.trim()) { + setStatus({ + type: 'error', + message: `${file.name} is empty or contains only whitespace` + }); + autoHideStatus(); + return; + } + + loadedFiles.push({ name: file.name, content: cleanContent }); } - // Calculate character counts - const addedChars = cleanContent.length; + const addedChars = loadedFiles.reduce((total, item) => total + item.content.length, 0); const totalChars = existingPromptLength + addedChars; - // Call parent callback - onFileLoaded(cleanContent, file.name); + for (const item of loadedFiles) { + onFileLoaded(item.content, item.name); + } // Show success message const charCountMsg = showCharCount @@ -100,8 +109,8 @@ function TextFileUploader({ setStatus({ type: 'success', - message: `✓ ${file.name} added${charCountMsg}${warningMsg}`, - filename: file.name + message: `✓ ${loadedFiles.length} file${loadedFiles.length === 1 ? '' : 's'} added${charCountMsg}${warningMsg}`, + filename: loadedFiles.map(item => item.name).join(', ') }); autoHideStatus(); @@ -183,10 +192,11 @@ function TextFileUploader({ <input ref={fileInputRef} type="file" - accept=".txt" + accept=".txt,.lean,text/plain,text/x-lean" + multiple onChange={handleFileChange} style={{ display: 'none' }} - aria-label="Upload text file to add to prompt" + aria-label="Upload .txt or .lean files to add as context" disabled={disabled || isLoading} /> @@ -195,7 +205,7 @@ function TextFileUploader({ onClick={handleButtonClick} disabled={disabled || isLoading} type="button" - title="Upload a .txt file to add its content to your prompt" + title="Upload .txt or .lean files to add as labeled context" > {isLoading ? ( <> @@ -204,7 +214,7 @@ function TextFileUploader({ </> ) : ( <> - 📎 Upload .txt File + 📎 Upload .txt or .lean files </> )} </button> diff --git a/frontend/src/components/WolframAlphaAccessModal.jsx b/frontend/src/components/WolframAlphaAccessModal.jsx index 2267847..9ab9275 100644 --- a/frontend/src/components/WolframAlphaAccessModal.jsx +++ b/frontend/src/components/WolframAlphaAccessModal.jsx @@ -116,7 +116,7 @@ export default function WolframAlphaAccessModal({ </div> <p className="settings-hint"> - Enable Wolfram Alpha tool calls for computational verification during compiler/autonomous construction mode. Use a Wolfram Alpha App ID with full results enabled. + Enable Wolfram Alpha tool calls for computational verification during compiler/autonomous construction mode. Use a Wolfram Alpha App ID with full results enabled. Wolfram Alpha is like giving your AI a very advanced calculator. </p> <label className="settings-checkbox-label settings-checkbox-label--stacked" style={{ marginBottom: '1rem' }}> diff --git a/frontend/src/components/WorkflowPanel.css b/frontend/src/components/WorkflowPanel.css index dc3de03..81b225f 100644 --- a/frontend/src/components/WorkflowPanel.css +++ b/frontend/src/components/WorkflowPanel.css @@ -1,3 +1,13 @@ +.solution-path-card { + width: 100%; display: flex; align-items: center; justify-content: space-between; + margin: 14px 0; padding: 12px; border: 1px solid rgba(139, 116, 255, .28); + border-radius: 10px; color: inherit; background: rgba(139, 116, 255, .07); + text-align: left; cursor: pointer; +} +.solution-path-card.available { border-color: rgba(91, 216, 155, .45); } +.solution-path-card span:first-child { display: grid; gap: 3px; } +.solution-path-card small { color: var(--text-secondary, #aaa); } + .workflow-panel { position: fixed; right: 0; @@ -284,6 +294,11 @@ padding: 0.75rem 1rem; background: var(--surface-0); border-bottom: 1px solid var(--border-subtle); + flex: 1; + min-height: 0; + display: flex; + flex-direction: column; + overflow-y: auto; } .token-stats-heading { @@ -423,3 +438,256 @@ .model-out { color: #e8a86a; } + +/* Assistant Memory Bank */ +.assistant-memory-bank { + margin-top: 0.75rem; + padding-top: 0.75rem; + border-top: 1px solid var(--border-subtle); + flex: 1; + min-height: 0; + display: flex; + flex-direction: column; +} + +.assistant-memory-bank__header { + display: flex; + justify-content: space-between; + gap: 0.75rem; + align-items: flex-start; + margin-bottom: 0.55rem; +} + +.assistant-memory-bank__title { + font-size: 0.75rem; + color: var(--text-muted); + text-transform: uppercase; + letter-spacing: 0.5px; +} + +.assistant-memory-bank__subtitle { + margin-top: 0.15rem; + font-size: 0.7rem; + color: var(--text-muted); + line-height: 1.35; +} + +.assistant-memory-bank__count { + flex-shrink: 0; + padding: 0.15rem 0.45rem; + border-radius: var(--radius-pill); + background: rgba(30, 255, 28, 0.08); + border: 1px solid rgba(30, 255, 28, 0.18); + color: #b7ffb6; + font-size: 0.68rem; + font-weight: 700; +} + +.assistant-memory-bank__empty { + padding: 0.55rem 0.6rem; + border: 1px dashed var(--border-subtle); + border-radius: var(--radius-sm); + background: var(--surface-1); + color: var(--text-muted); + font-size: 0.72rem; + line-height: 1.4; +} + +.assistant-proof-list { + display: flex; + flex-direction: column; + gap: 0.45rem; + max-height: min(58vh, 620px); + overflow-y: auto; + padding-right: 0.2rem; +} + +.assistant-proof-list::-webkit-scrollbar { + width: 4px; +} + +.assistant-proof-list::-webkit-scrollbar-thumb { + background: #444; + border-radius: 2px; +} + +.assistant-proof-tile { + width: 100%; + text-align: left; + padding: 0.55rem 0.6rem; + border: 1px solid var(--border-subtle); + border-left-width: 3px; + border-radius: var(--radius-sm); + background: var(--surface-1); + color: var(--text-secondary); + cursor: pointer; + transition: border-color var(--transition-fast), background var(--transition-fast), transform var(--transition-fast); +} + +.assistant-proof-tile:hover, +.assistant-proof-tile.selected { + background: var(--surface-2); + border-color: var(--border-strong); + transform: translateX(-1px); +} + +.assistant-proof-tile--platinum { + border-left-color: #e2e8f0; + box-shadow: inset 3px 0 12px rgba(226, 232, 240, 0.14); +} + +.assistant-proof-tile--gold { + border-left-color: #ffd65c; +} + +.assistant-proof-tile--silver { + border-left-color: #c0c0c0; +} + +.assistant-proof-tile--bronze { + border-left-color: #cd7f32; +} + +.assistant-proof-tile--known { + border-left-color: #666; +} + +.assistant-proof-tile__topline { + display: flex; + justify-content: space-between; + align-items: center; + gap: 0.45rem; + margin-bottom: 0.25rem; +} + +.assistant-proof-tile__tier, +.assistant-proof-tile__source { + font-size: 0.62rem; + font-weight: 800; + text-transform: uppercase; + letter-spacing: 0.05em; +} + +.assistant-proof-tile__tier { + color: #b7ffb6; +} + +.assistant-proof-tile__source { + color: var(--text-muted); +} + +.assistant-proof-tile__name { + color: var(--text-primary); + font-family: 'Courier New', Courier, monospace; + font-size: 0.78rem; + font-weight: 700; + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; +} + +.assistant-proof-tile__statement { + margin-top: 0.25rem; + color: var(--text-secondary); + font-size: 0.72rem; + line-height: 1.35; +} + +.assistant-proof-tile__hint { + margin-top: 0.35rem; + color: var(--text-muted); + font-size: 0.68rem; + line-height: 1.35; + font-style: italic; +} + +.assistant-proof-preview { + margin-top: 0.65rem; + padding: 0.65rem; + border: 1px solid rgba(30, 255, 28, 0.16); + border-radius: var(--radius-sm); + background: rgba(0, 0, 0, 0.22); +} + +.assistant-proof-preview__header { + display: flex; + justify-content: space-between; + gap: 0.5rem; + align-items: flex-start; + margin-bottom: 0.5rem; +} + +.assistant-proof-preview__label { + color: var(--text-muted); + font-size: 0.62rem; + font-weight: 800; + text-transform: uppercase; + letter-spacing: 0.05em; +} + +.assistant-proof-preview__title { + margin-top: 0.15rem; + color: var(--text-primary); + font-size: 0.78rem; + font-weight: 700; + word-break: break-word; +} + +.assistant-proof-preview__close { + flex-shrink: 0; + border: 1px solid var(--border-subtle); + border-radius: var(--radius-sm); + background: var(--surface-2); + color: var(--text-muted); + font-size: 0.65rem; + cursor: pointer; + padding: 0.2rem 0.4rem; +} + +.assistant-proof-preview__close:hover { + color: var(--text-primary); + border-color: var(--border-strong); +} + +.assistant-proof-preview__meta { + display: flex; + flex-wrap: wrap; + gap: 0.25rem; + margin-bottom: 0.45rem; +} + +.assistant-proof-preview__meta span { + padding: 0.14rem 0.4rem; + border-radius: var(--radius-pill); + background: rgba(30, 255, 28, 0.07); + color: #b7ffb6; + border: 1px solid rgba(30, 255, 28, 0.14); + font-size: 0.62rem; +} + +.assistant-proof-preview__statement, +.assistant-proof-preview__description, +.assistant-proof-preview__loading, +.assistant-proof-preview__error { + margin: 0.35rem 0 0; + color: var(--text-secondary); + font-size: 0.72rem; + line-height: 1.4; +} + +.assistant-proof-preview__error { + color: #ff9f9f; +} + +.assistant-proof-preview__code { + margin: 0.5rem 0 0; + max-height: 220px; + overflow: auto; + padding: 0.5rem; + border-radius: var(--radius-sm); + background: rgba(0, 0, 0, 0.35); + color: #d8d8e5; + font-size: 0.65rem; + line-height: 1.35; + white-space: pre-wrap; +} diff --git a/frontend/src/components/WorkflowPanel.jsx b/frontend/src/components/WorkflowPanel.jsx index 90fa01e..094318a 100644 --- a/frontend/src/components/WorkflowPanel.jsx +++ b/frontend/src/components/WorkflowPanel.jsx @@ -1,12 +1,24 @@ import React, { useState, useEffect, useCallback, useMemo, useRef } from 'react'; import { websocket } from '../services/websocket'; -import { boostAPI, workflowAPI } from '../services/api'; +import { boostAPI, proofSearchAPI, workflowAPI } from '../services/api'; +import { + classifyProofNovelty, + formatProofProvenance, + getCanonicalProofIdentity, + sanitizeDomId, +} from '../utils/proofPresentation'; +import { + getSolutionPathEmptyLabel, + isSolutionPathSnapshotAtLeast, + solutionPathEventMatches, +} from '../utils/solutionPathPresentation'; import './WorkflowPanel.css'; const AUTO_OPEN_DELAY_SECONDS = 600; const BOOST_CATEGORY_GROUPS = ['Aggregator', 'Compiler', 'Autonomous', 'Proof Solver']; const MAX_SUBMITTERS = 10; const WORKFLOW_PANEL_COLLAPSED_KEY = 'workflow_panel_collapsed'; +const ASSISTANT_MAX_RESULTS = 7; const readStoredCollapsedState = () => { if (typeof window === 'undefined') return true; @@ -150,9 +162,35 @@ const formatTime = (totalSeconds) => { return `${String(h).padStart(2, '0')}h ${String(m).padStart(2, '0')}m ${String(s).padStart(2, '0')}s`; }; +const truncateText = (text, maxLength = 140) => { + const normalized = String(text || '').replace(/\s+/g, ' ').trim(); + if (normalized.length <= maxLength) return normalized; + return `${normalized.slice(0, maxLength).trim()}...`; +}; + +const formatAssistantTierLabel = (support = {}) => classifyProofNovelty(support).shortLabel; + +const getAssistantTileClass = (support = {}) => classifyProofNovelty(support).tileClass; + +const formatAssistantSource = (support = {}) => ( + [ + support.corpus, + support.corpus_scope, + support.session_id ? `session ${support.session_id}` : '', + ].filter(Boolean).join(' · ') || 'proof history' +); + +const normalizeAssistantPack = (pack = {}) => ({ + ...pack, + results: Array.isArray(pack.results) ? pack.results.slice(0, ASSISTANT_MAX_RESULTS) : [], +}); + export default function WorkflowPanel({ isRunning, onOpenBoostSettings, + onOpenAssistantProof, + onOpenSolutionPath, + onSolutionPathSnapshotChange, collapsed: controlledCollapsed, onCollapseChange, }) { @@ -177,6 +215,17 @@ export default function WorkflowPanel({ const [localElapsed, setLocalElapsed] = useState(0); const lastSyncRef = useRef(Date.now()); const hasElapsedSyncRef = useRef(false); + const [assistantMemoryPack, setAssistantMemoryPack] = useState(null); + const [solutionPath, setSolutionPath] = useState(null); + const solutionPathRef = useRef(null); + const solutionPathRequestRef = useRef(0); + const [assistantMemoryError, setAssistantMemoryError] = useState(''); + const [selectedAssistantProof, setSelectedAssistantProof] = useState(null); + const [assistantProofDetail, setAssistantProofDetail] = useState(null); + const [assistantProofLoading, setAssistantProofLoading] = useState(false); + const assistantPackGenerationRef = useRef(0); + const assistantDetailGenerationRef = useRef(0); + const mountedRef = useRef(true); // Auto-open: pop open exactly once, 10 minutes after user presses Start. // No persistence. Resets every time isRunning goes true. @@ -202,6 +251,55 @@ export default function WorkflowPanel({ } }, [isRunning]); + const fetchSolutionPath = useCallback(async () => { + const requestSequence = ++solutionPathRequestRef.current; + try { + const snapshot = await workflowAPI.getSolutionPath(); + if (!mountedRef.current || requestSequence !== solutionPathRequestRef.current) return null; + const current = solutionPathRef.current; + if (isSolutionPathSnapshotAtLeast(snapshot, current)) { + solutionPathRef.current = snapshot; + setSolutionPath(snapshot); + onSolutionPathSnapshotChange?.(snapshot); + } + return snapshot; + } catch (error) { + if (!mountedRef.current || requestSequence !== solutionPathRequestRef.current) return null; + console.debug('Failed to fetch solution path:', error); + return null; + } + }, [onSolutionPathSnapshotChange]); + + useEffect(() => { + fetchSolutionPath(); + }, [fetchSolutionPath, isRunning]); + + useEffect(() => { + const handleChanged = (data = {}) => { + const current = solutionPathRef.current; + if (!solutionPathEventMatches(data, current, current?.mode || '')) return; + if ( + current?.run_id === data.run_id + && Number(data.lifecycle_generation || 0) + === Number(current?.lifecycle_generation || 0) + && Number(data.revision || 0) < Number(current?.revision || 0) + ) return; + fetchSolutionPath(); + }; + const events = [ + 'solution_path_activated', + 'solution_path_proposal_queued', + 'solution_path_proposal_reviewing', + 'solution_path_updated', + 'solution_path_proposal_rejected', + 'solution_path_proposal_retry_queued', + 'solution_path_proposal_user_repair_required', + 'solution_path_proposal_resumed', + ]; + events.forEach((eventName) => websocket.on(eventName, handleChanged)); + return () => events.forEach((eventName) => websocket.off(eventName, handleChanged)); + }, [fetchSolutionPath]); + useEffect(() => { if (!isRunning || hasPoppedThisSession.current) return; if (localElapsed >= AUTO_OPEN_DELAY_SECONDS) { @@ -235,6 +333,46 @@ export default function WorkflowPanel({ return () => clearInterval(interval); }, [fetchBoostStatus]); + const fetchAssistantMemoryPack = useCallback(async () => { + const generation = ++assistantPackGenerationRef.current; + try { + const pack = await proofSearchAPI.getAssistantLatestPack(); + if (!mountedRef.current || generation !== assistantPackGenerationRef.current) return; + if (pack?.enabled === false) { + setAssistantMemoryPack(normalizeAssistantPack({ ...pack, results: [] })); + setSelectedAssistantProof(null); + setAssistantProofDetail(null); + setAssistantMemoryError(pack.disabled_reason || 'Session History Memory is disabled.'); + return; + } + setAssistantMemoryPack(normalizeAssistantPack(pack)); + setAssistantMemoryError(pack.disabled_reason || ''); + } catch (error) { + if (!mountedRef.current || generation !== assistantPackGenerationRef.current) return; + setAssistantMemoryError(error.message || 'Assistant memory unavailable'); + } + }, []); + + useEffect(() => { + mountedRef.current = true; + return () => { + mountedRef.current = false; + assistantPackGenerationRef.current += 1; + assistantDetailGenerationRef.current += 1; + solutionPathRequestRef.current += 1; + }; + }, []); + + useEffect(() => { + fetchAssistantMemoryPack(); + }, [fetchAssistantMemoryPack]); + + useEffect(() => { + if (!collapsed) { + fetchAssistantMemoryPack(); + } + }, [collapsed, fetchAssistantMemoryPack]); + useEffect(() => { if (!isEditingBoostNext) { setBoostNextInput(boostNextCount > 0 ? boostNextCount.toString() : ''); @@ -310,6 +448,41 @@ export default function WorkflowPanel({ return () => websocket.off('token_usage_updated', handleTokenUpdate); }, []); + useEffect(() => { + const handleAssistantPackUpdated = (data) => { + if (data?.enabled === false) { + setAssistantMemoryPack(normalizeAssistantPack({ ...data, results: [] })); + setSelectedAssistantProof(null); + setAssistantProofDetail(null); + setAssistantMemoryError(data.disabled_reason || 'Session History Memory is disabled.'); + } else if (Array.isArray(data?.results)) { + setAssistantMemoryPack(normalizeAssistantPack({ ...data, has_pack: true, enabled: true })); + setAssistantMemoryError(''); + } else { + fetchAssistantMemoryPack(); + } + }; + const handleAssistantPackFailed = (data = {}) => { + setAssistantMemoryError(data.error_message || data.reason || 'Assistant memory selection failed'); + }; + const handleAssistantMemoryUnavailable = (data = {}) => { + setAssistantMemoryPack((previous) => previous ? { ...previous, results: [] } : previous); + setSelectedAssistantProof(null); + setAssistantProofDetail(null); + setAssistantMemoryError(data.reason || 'Assistant memory found no external proof history yet.'); + }; + websocket.on('assistant_proof_pack_updated', handleAssistantPackUpdated); + websocket.on('assistant_proof_pack_failed', handleAssistantPackFailed); + websocket.on('assistant_proof_memory_unavailable', handleAssistantMemoryUnavailable); + websocket.on('assistant_proof_memory_shutdown', handleAssistantMemoryUnavailable); + return () => { + websocket.off('assistant_proof_pack_updated', handleAssistantPackUpdated); + websocket.off('assistant_proof_pack_failed', handleAssistantPackFailed); + websocket.off('assistant_proof_memory_unavailable', handleAssistantMemoryUnavailable); + websocket.off('assistant_proof_memory_shutdown', handleAssistantMemoryUnavailable); + }; + }, [fetchAssistantMemoryPack]); + // Local 1-second timer tick for smooth elapsed display useEffect(() => { if (!isRunning) return; @@ -414,6 +587,40 @@ export default function WorkflowPanel({ setPanelCollapsed(!collapsed); }; + const handleAssistantProofClick = async (support) => { + const identity = getCanonicalProofIdentity(support); + const generation = ++assistantDetailGenerationRef.current; + setSelectedAssistantProof(support); + setAssistantProofDetail(null); + setAssistantProofLoading(true); + onOpenAssistantProof?.(support); + try { + const hydrated = await proofSearchAPI.getProof(support.corpus, support.proof_id, { + searchId: support.search_id || null, + runId: support.run_id || null, + sessionId: support.session_id || null, + }); + if ( + mountedRef.current + && generation === assistantDetailGenerationRef.current + && getCanonicalProofIdentity(support) === identity + ) setAssistantProofDetail(hydrated); + } catch (error) { + if (!mountedRef.current || generation !== assistantDetailGenerationRef.current) return; + setAssistantProofDetail({ + ...support, + hydration_error: error.message || 'Proof history details could not be opened.', + }); + } finally { + if (mountedRef.current && generation === assistantDetailGenerationRef.current) { + setAssistantProofLoading(false); + } + } + }; + + const assistantResults = assistantMemoryPack?.results || []; + const selectedProofView = assistantProofDetail || selectedAssistantProof; + // REMOVED: Conditional rendering that hid panel when no workflow running // WorkflowPanel is now ETERNAL - always visible for boost controls // User can access boost configuration at any time, not just during active research @@ -582,6 +789,175 @@ export default function WorkflowPanel({ )} </div> )} + + {(solutionPath?.enabled || solutionPath?.acceptance_count >= 5) && ( + <button + type="button" + className={`solution-path-card ${solutionPath?.enabled ? 'available' : ''}`} + onClick={async () => { + const snapshot = await fetchSolutionPath(); + onOpenSolutionPath?.(snapshot || solutionPath); + }} + aria-label="Open current solution path" + > + <span> + <strong>Solution Path</strong> + <small> + {solutionPath?.repair_required_proposals > 0 + ? `${solutionPath.repair_required_proposals} solution-path update(s) need attention` + : solutionPath?.reviewing_proposals > 0 + ? 'Solution-path update under review' + : solutionPath?.queued_proposals > 0 + ? `${solutionPath.queued_proposals} solution-path update(s) queued` + : solutionPath?.enabled && (solutionPath.steps?.length || 0) > 0 + ? `${solutionPath.steps?.length || 0} steps · revision ${solutionPath.revision || 1}` + : getSolutionPathEmptyLabel(solutionPath)} + </small> + </span> + <span aria-hidden="true">›</span> + </button> + )} + + <div className="assistant-memory-bank"> + <div className="assistant-memory-bank__header"> + <div> + <div className="assistant-memory-bank__title">Assistant Memory Bank</div> + <div className="assistant-memory-bank__subtitle"> + Latest proof supports retrieved by Assistant memory. + </div> + </div> + {assistantMemoryPack?.has_pack && ( + <span className="assistant-memory-bank__count"> + {assistantResults.length}/{assistantMemoryPack.max_result_count || ASSISTANT_MAX_RESULTS} + </span> + )} + </div> + + {assistantMemoryError && assistantResults.length === 0 && ( + <div className="assistant-memory-bank__empty">{assistantMemoryError}</div> + )} + + {!assistantMemoryError && assistantResults.length === 0 && ( + <div className="assistant-memory-bank__empty"> + No Assistant proof pack has been retrieved yet. + </div> + )} + + {assistantResults.length > 0 && ( + <div className="assistant-proof-list"> + {assistantResults.map((support, index) => { + const supportKey = getCanonicalProofIdentity(support, { includeIndex: true, index }); + const supportIdentity = getCanonicalProofIdentity(support); + const selectedKey = selectedAssistantProof ? getCanonicalProofIdentity(selectedAssistantProof) : ''; + const isSelected = selectedKey === supportIdentity; + const detailId = sanitizeDomId(supportIdentity, 'assistant-proof-details'); + const provenance = formatProofProvenance(support); + return ( + <button + type="button" + key={supportKey} + className={`assistant-proof-tile ${getAssistantTileClass(support)} ${isSelected ? 'selected' : ''}`} + onClick={() => handleAssistantProofClick(support)} + title="Open proof history preview" + aria-expanded={isSelected} + aria-controls={detailId} + > + <div className="assistant-proof-tile__topline"> + <span className="assistant-proof-tile__tier">{formatAssistantTierLabel(support)}</span> + <span className="assistant-proof-tile__source">{support.corpus}</span> + </div> + <div className="assistant-proof-tile__name"> + {support.theorem_name || support.proof_id || 'Untitled proof'} + </div> + <div className="assistant-proof-tile__statement"> + {truncateText(support.theorem_statement, 128)} + </div> + <div className="assistant-proof-tile__hint"> + {[ + ...provenance.lanes.map((lane) => `Lane: ${lane}`), + provenance.runId && `Run: ${provenance.runId}`, + provenance.sessionId && `Session: ${provenance.sessionId}`, + provenance.source && `Source: ${provenance.source}`, + provenance.omitted > 0 && `+${provenance.omitted} omitted lineage records`, + ].filter(Boolean).join(' · ')} + </div> + {(support.relevance_reason || support.transfer_hint) && ( + <div className="assistant-proof-tile__hint"> + {truncateText(support.relevance_reason || support.transfer_hint, 110)} + </div> + )} + </button> + ); + })} + </div> + )} + + <div + id={sanitizeDomId( + selectedAssistantProof ? getCanonicalProofIdentity(selectedAssistantProof) : '', + 'assistant-proof-details' + )} + className="assistant-proof-preview" + hidden={!selectedAssistantProof} + > + {selectedAssistantProof && ( + <> + <div className="assistant-proof-preview__header"> + <div> + <div className="assistant-proof-preview__label">Proof History Preview</div> + <div className="assistant-proof-preview__title"> + {selectedProofView?.theorem_name || selectedProofView?.display_title || selectedProofView?.proof_id} + </div> + </div> + <button + type="button" + className="assistant-proof-preview__close" + onClick={() => { + assistantDetailGenerationRef.current += 1; + setSelectedAssistantProof(null); + setAssistantProofDetail(null); + setAssistantProofLoading(false); + }} + > + Close + </button> + </div> + {assistantProofLoading ? ( + <div className="assistant-proof-preview__loading">Opening proof history...</div> + ) : ( + <> + <div className="assistant-proof-preview__meta"> + <span>{formatAssistantSource(selectedProofView)}</span> + {selectedProofView?.proof_id && <span>{selectedProofView.proof_id}</span>} + <span>{formatAssistantTierLabel(selectedProofView)}</span> + {selectedProofView?.run_id && <span>Run: {selectedProofView.run_id}</span>} + {selectedProofView?.session_id && <span>Session: {selectedProofView.session_id}</span>} + {selectedProofView?.source_type && <span>Source: {selectedProofView.source_type}{selectedProofView.source_id ? `/${selectedProofView.source_id}` : ''}</span>} + {(selectedProofView?.retrieval_lanes || []).map((lane) => <span key={lane}>Lane: {lane}</span>)} + {formatProofProvenance(selectedProofView).omitted > 0 && <span>+{formatProofProvenance(selectedProofView).omitted} omitted lineage records</span>} + </div> + <p className="assistant-proof-preview__statement"> + {selectedProofView?.theorem_statement || 'No theorem statement available.'} + </p> + {(selectedProofView?.proof_description || selectedProofView?.formal_sketch) && ( + <p className="assistant-proof-preview__description"> + {selectedProofView.proof_description || selectedProofView.formal_sketch} + </p> + )} + {selectedProofView?.hydration_error && ( + <div className="assistant-proof-preview__error">{selectedProofView.hydration_error}</div> + )} + {selectedProofView?.lean_code && ( + <pre className="assistant-proof-preview__code"> + {truncateText(selectedProofView.lean_code, 900)} + </pre> + )} + </> + )} + </> + )} + </div> + </div> </div> </> )} diff --git a/frontend/src/components/WorkflowPanel.test.jsx b/frontend/src/components/WorkflowPanel.test.jsx new file mode 100644 index 0000000..7a39ef4 --- /dev/null +++ b/frontend/src/components/WorkflowPanel.test.jsx @@ -0,0 +1,304 @@ +import React from 'react'; +import { act, render, screen, waitFor } from '@testing-library/react'; +import userEvent from '@testing-library/user-event'; +import WorkflowPanel from './WorkflowPanel'; +import { proofSearchAPI, workflowAPI } from '../services/api'; + +const listeners = new Map(); + +vi.mock('../services/websocket', () => ({ + websocket: { + on: vi.fn((event, callback) => { + if (!listeners.has(event)) listeners.set(event, []); + listeners.get(event).push(callback); + }), + off: vi.fn((event, callback) => { + const callbacks = listeners.get(event) || []; + const index = callbacks.indexOf(callback); + if (index >= 0) callbacks.splice(index, 1); + }), + }, +})); + +vi.mock('../services/api', () => ({ + boostAPI: { + getStatus: vi.fn().mockResolvedValue({ + success: true, + status: { + enabled: false, + boost_next_count: 0, + boosted_categories: [], + boost_always_prefer: false, + }, + }), + getCategories: vi.fn().mockResolvedValue({ success: true, categories: [] }), + setNextCount: vi.fn(), + setAlwaysPrefer: vi.fn(), + toggleCategory: vi.fn(), + }, + workflowAPI: { + getTokenStats: vi.fn().mockResolvedValue({ + success: true, + total_input: 10, + total_output: 20, + by_model: {}, + elapsed_seconds: 30, + }), + getPredictions: vi.fn().mockResolvedValue({ success: true, mode: 'idle', tasks: [] }), + getSolutionPath: vi.fn().mockResolvedValue({ + success: true, + enabled: false, + mode: 'idle', + run_id: null, + revision: null, + steps: [], + queued_proposals: 0, + reviewing_proposals: 0, + }), + }, + proofSearchAPI: { + getAssistantLatestPack: vi.fn(), + getProof: vi.fn(), + }, +})); + +const assistantPack = { + enabled: true, + has_pack: true, + target_hash: 'target_hash', + result_count: 1, + max_result_count: 7, + results: [ + { + search_id: 'manual:proof_1', + corpus: 'manual', + corpus_scope: 'history', + proof_id: 'proof_1', + session_id: 'manual_run_1', + theorem_name: 'Memory.Helper', + theorem_statement: 'theorem helper : True', + proof_description: 'A hydrated proof support.', + novelty_tier: 'mathematical_discovery', + novelty_reasoning: 'Useful discovery.', + relevance_reason: 'Matches the active target.', + lean_code: '', + has_hydrated_code: true, + }, + ], +}; + +beforeEach(() => { + vi.clearAllMocks(); + listeners.clear(); + proofSearchAPI.getAssistantLatestPack.mockResolvedValue(assistantPack); + proofSearchAPI.getProof.mockResolvedValue({ + ...assistantPack.results[0], + proof_description: 'Hydrated proof history detail.', + lean_code: 'theorem helper : True := by trivial', + }); + window.localStorage.clear(); + window.localStorage.setItem('workflow_panel_collapsed', 'false'); +}); + +test('renders Assistant Memory Bank proof tiles and opens hydrated proof preview', async () => { + const user = userEvent.setup(); + const onOpenAssistantProof = vi.fn(); + + render( + <WorkflowPanel + isRunning={false} + onOpenBoostSettings={vi.fn()} + onOpenAssistantProof={onOpenAssistantProof} + /> + ); + + expect(await screen.findByText('Assistant Memory Bank')).toBeInTheDocument(); + const tile = await screen.findByRole('button', { name: /Memory.Helper/i }); + expect(tile.className).toContain('assistant-proof-tile--gold'); + expect(screen.getByText('1/7')).toBeInTheDocument(); + + await user.click(tile); + + expect(onOpenAssistantProof).toHaveBeenCalledWith(expect.objectContaining({ proof_id: 'proof_1' })); + await waitFor(() => { + expect(proofSearchAPI.getProof).toHaveBeenCalledWith('manual', 'proof_1', { + searchId: 'manual:proof_1', + runId: null, + sessionId: 'manual_run_1', + }); + }); + expect(await screen.findByText(/Hydrated proof history detail/i)).toBeInTheDocument(); + expect(screen.getByText(/theorem helper : True := by trivial/i)).toBeInTheDocument(); +}); + +test('shows Assistant memory disabled reason from latest-pack status', async () => { + proofSearchAPI.getAssistantLatestPack.mockResolvedValue({ + enabled: false, + has_pack: false, + results: [], + disabled_reason: 'Session History Memory is disabled.', + }); + + render( + <WorkflowPanel + isRunning={false} + onOpenBoostSettings={vi.fn()} + onOpenAssistantProof={vi.fn()} + /> + ); + + expect(await screen.findByText('Session History Memory is disabled.')).toBeInTheDocument(); +}); + +test('disabled latest-pack response clears stale Assistant proof tiles', async () => { + proofSearchAPI.getAssistantLatestPack.mockResolvedValue(assistantPack); + + render( + <WorkflowPanel + isRunning={false} + onOpenBoostSettings={vi.fn()} + onOpenAssistantProof={vi.fn()} + collapsed={false} + /> + ); + + expect(await screen.findByText('Memory.Helper')).toBeInTheDocument(); + act(() => { + listeners.get('assistant_proof_pack_updated')?.forEach((callback) => callback({ + enabled: false, + has_pack: false, + results: [], + disabled_reason: 'Session History Memory is disabled.', + })); + }); + await waitFor(() => { + expect(screen.queryByText('Memory.Helper')).not.toBeInTheDocument(); + }); + expect(await screen.findByText('Session History Memory is disabled.')).toBeInTheDocument(); +}); + +test('refreshes queued solution-path state and ignores stale revision events', async () => { + workflowAPI.getSolutionPath + .mockResolvedValueOnce({ + success: true, + enabled: true, + mode: 'autonomous', + run_id: 'run-1', + acceptance_count: 5, + revision: 3, + steps: [{ step_id: 'one', title: 'First', status: 'active' }], + queued_proposals: 0, + reviewing_proposals: 0, + }) + .mockResolvedValueOnce({ + success: true, + enabled: true, + mode: 'autonomous', + run_id: 'run-1', + acceptance_count: 5, + revision: 3, + steps: [{ step_id: 'one', title: 'First', status: 'active' }], + queued_proposals: 1, + reviewing_proposals: 0, + }); + + render( + <WorkflowPanel + isRunning + onOpenBoostSettings={vi.fn()} + onOpenAssistantProof={vi.fn()} + collapsed={false} + /> + ); + expect(await screen.findByText(/1 steps · revision 3/i)).toBeInTheDocument(); + + act(() => { + listeners.get('solution_path_proposal_queued')?.forEach((callback) => callback({ + run_id: 'run-1', + revision: 3, + })); + }); + await waitFor(() => expect(workflowAPI.getSolutionPath).toHaveBeenCalledTimes(2)); + expect(screen.getByText(/1 solution-path update\(s\) queued/i)).toBeInTheDocument(); + + act(() => { + listeners.get('solution_path_updated')?.forEach((callback) => callback({ + run_id: 'run-1', + revision: 2, + })); + }); + expect(workflowAPI.getSolutionPath).toHaveBeenCalledTimes(2); +}); + +test('hides the solution path before five accepted ideas', async () => { + workflowAPI.getSolutionPath.mockResolvedValue({ + success: true, + enabled: false, + ownership: 'active', + mode: 'aggregator', + run_id: 'run-pre-five', + acceptance_count: 4, + steps: [], + }); + + render( + <WorkflowPanel + isRunning + onOpenBoostSettings={vi.fn()} + onOpenAssistantProof={vi.fn()} + collapsed={false} + /> + ); + + await waitFor(() => expect(workflowAPI.getSolutionPath).toHaveBeenCalled()); + expect(screen.queryByRole('button', { name: /open current solution path/i })).not.toBeInTheDocument(); +}); + +test('ignores an older overlapping solution-path request', async () => { + let resolveFirst; + workflowAPI.getSolutionPath + .mockImplementationOnce(() => new Promise((resolve) => { resolveFirst = resolve; })) + .mockResolvedValueOnce({ + success: true, + enabled: true, + ownership: 'active', + mode: 'autonomous', + run_id: 'new-run', + lifecycle_generation: 2, + acceptance_count: 5, + revision: 2, + steps: [{ step_id: 'new', title: 'New route', status: 'active' }], + }); + + const { rerender } = render( + <WorkflowPanel + isRunning={false} + onOpenBoostSettings={vi.fn()} + onOpenAssistantProof={vi.fn()} + collapsed={false} + /> + ); + rerender( + <WorkflowPanel + isRunning + onOpenBoostSettings={vi.fn()} + onOpenAssistantProof={vi.fn()} + collapsed={false} + /> + ); + + expect(await screen.findByText(/1 steps · revision 2/i)).toBeInTheDocument(); + await act(async () => { + resolveFirst({ + success: true, + enabled: false, + ownership: 'resumable', + mode: 'aggregator', + run_id: 'old-run', + acceptance_count: 5, + steps: [], + }); + }); + expect(screen.getByText(/1 steps · revision 2/i)).toBeInTheDocument(); +}); + diff --git a/frontend/src/components/aggregator/AggregatorInterface.jsx b/frontend/src/components/aggregator/AggregatorInterface.jsx index df2a178..1740f75 100644 --- a/frontend/src/components/aggregator/AggregatorInterface.jsx +++ b/frontend/src/components/aggregator/AggregatorInterface.jsx @@ -1,6 +1,7 @@ import React, { useState, useEffect } from 'react'; import { api } from '../../services/api'; import TextFileUploader from '../TextFileUploader'; +import '../autonomous/AutonomousResearch.css'; import '../settings-common.css'; export default function AggregatorInterface({ @@ -14,8 +15,8 @@ export default function AggregatorInterface({ }) { const [isRunning, setIsRunning] = useState(false); const [status, setStatus] = useState(null); - const [uploadedFiles, setUploadedFiles] = useState([]); const lmStudioEnabled = capabilities?.lmStudioEnabled !== false; + const attachedFiles = config.uploadedFiles || []; useEffect(() => { fetchStatus(); @@ -38,27 +39,45 @@ export default function AggregatorInterface({ const handleFileUpload = async (event) => { const files = Array.from(event.target.files); + event.target.value = ''; for (const file of files) { try { const result = await api.uploadFile(file); - setUploadedFiles(prev => [...prev, result.path]); setConfig(prev => ({ ...prev, - uploadedFiles: [...prev.uploadedFiles, result.path] + uploadedFiles: Array.from(new Set([...(prev.uploadedFiles || []), result.path])) })); } catch (error) { console.error('Failed to upload file:', error); - alert(`Failed to upload ${file.name}`); + alert(`Failed to upload ${file.name}: ${error.details || error.message}`); } } }; - const handleTextFileLoaded = (content) => { - // Append to existing prompt with separator - const separator = config.userPrompt.trim() ? '\n\n' : ''; - const newPrompt = config.userPrompt + separator + content; - setConfig({ ...config, userPrompt: newPrompt }); + const handleTextFileLoaded = (content, filename = 'uploaded-file') => { + const isLeanFile = filename.toLowerCase().endsWith('.lean'); + const labeledContext = `[UPLOADED ${isLeanFile ? 'LEAN FILE' : 'TEXT FILE'}: ${filename}]\n${content}`; + setConfig(prev => { + const separator = prev.userPrompt.trim() ? '\n\n' : ''; + return { + ...prev, + userPrompt: prev.userPrompt + separator + labeledContext + }; + }); + }; + + const handleRemoveUploadedFile = async (filePath) => { + setConfig(prev => ({ + ...prev, + uploadedFiles: (prev.uploadedFiles || []).filter(path => path !== filePath) + })); + try { + await api.deleteUploadedFile(filePath); + } catch (error) { + console.error('Failed to remove uploaded file:', error); + alert(`Removed ${filePath} from this run, but failed to delete it from upload storage: ${error.details || error.message}`); + } }; const handleStart = async () => { @@ -166,60 +185,90 @@ export default function AggregatorInterface({ </div> </div> - <div className="form-group"> - <label>User Prompt *</label> - <textarea - value={config.userPrompt} - onChange={(e) => setConfig({ ...config, userPrompt: e.target.value })} - placeholder='Be descriptive, this prompt should direct an open-ended brainstorming question, i.e. "Tell me all of the reasons we both should or should not be able to mathematically \"square the circle\"."' - disabled={isRunning} - /> - <TextFileUploader - onFileLoaded={handleTextFileLoaded} - disabled={isRunning} - maxSizeMB={5} - showCharCount={true} - confirmIfNotEmpty={true} - existingPromptLength={config.userPrompt.length} - /> + <div className="form-group prompt-composer-section"> + <div className="prompt-composer"> + <label htmlFor="aggregator-user-prompt">User Prompt *</label> + <textarea + id="aggregator-user-prompt" + value={config.userPrompt} + onChange={(e) => setConfig({ ...config, userPrompt: e.target.value })} + placeholder='Be descriptive, this prompt should direct an open-ended brainstorming question, i.e. "Tell me all of the reasons we both should or should not be able to mathematically \"square the circle\"."' + disabled={isRunning} + rows={5} + /> + <div className="prompt-composer-actions"> + <TextFileUploader + onFileLoaded={handleTextFileLoaded} + disabled={isRunning} + maxSizeMB={5} + showCharCount={true} + confirmIfNotEmpty={true} + existingPromptLength={config.userPrompt.length} + /> + {!isRunning ? ( + <button + type="button" + onClick={handleStart} + className="btn-start prompt-composer-primary" + disabled={anyWorkflowRunning && !isRunning} + > + Start Aggregator + </button> + ) : ( + <button type="button" onClick={handleStop} className="btn-stop"> + Stop Aggregator + </button> + )} + </div> + </div> + {developerModeEnabled && ( + <div className="prompt-composer-options"> + <label className="settings-checkbox-label prompt-composer-option"> + <input + type="checkbox" + checked={Boolean(config.creativityEmphasisBoostEnabled)} + onChange={(e) => setConfig({ ...config, creativityEmphasisBoostEnabled: e.target.checked })} + disabled={isRunning} + /> + Creativity Emphasis Boost + </label> + </div> + )} </div> <div className="form-group"> - <label>Upload Files (optional)</label> + <label htmlFor="aggregator-context-file-upload">Attach Context Files (optional)</label> <input + id="aggregator-context-file-upload" type="file" onChange={handleFileUpload} multiple + accept=".txt,.lean,text/plain,text/x-lean" disabled={isRunning} /> - {uploadedFiles.length > 0 && ( - <div style={{ marginTop: '0.5rem', color: '#4CAF50' }}> - Uploaded {uploadedFiles.length} file(s) + <small className="settings-hint"> + Attached .txt and .lean files are kept as separate context sources and use direct-first/RAG allocation. + </small> + {attachedFiles.length > 0 && ( + <div className="attached-file-list" aria-label="Attached context files"> + {attachedFiles.map((filePath) => ( + <span className="attached-file-chip" key={filePath}> + <span className="attached-file-name">{filePath}</span> + <button + type="button" + className="attached-file-remove" + onClick={() => handleRemoveUploadedFile(filePath)} + disabled={isRunning} + aria-label={`Remove ${filePath}`} + > + × + </button> + </span> + ))} </div> )} </div> - <div className="button-group"> - {!isRunning ? ( - <button onClick={handleStart} disabled={anyWorkflowRunning && !isRunning}> - Start Aggregator - </button> - ) : ( - <button onClick={handleStop} className="danger">Stop Aggregator</button> - )} - {developerModeEnabled && ( - <label className="settings-checkbox-label"> - <input - type="checkbox" - checked={Boolean(config.creativityEmphasisBoostEnabled)} - onChange={(e) => setConfig({ ...config, creativityEmphasisBoostEnabled: e.target.checked })} - disabled={isRunning} - /> - Creativity Emphasis Boost - </label> - )} - </div> - {status && ( <div className="grid-3"> <div className="metric-card"> diff --git a/frontend/src/components/aggregator/AggregatorInterface.test.jsx b/frontend/src/components/aggregator/AggregatorInterface.test.jsx index ccec6c0..cf1514a 100644 --- a/frontend/src/components/aggregator/AggregatorInterface.test.jsx +++ b/frontend/src/components/aggregator/AggregatorInterface.test.jsx @@ -1,3 +1,4 @@ +import React from 'react'; import { fireEvent, render, screen, waitFor } from '@testing-library/react'; import { beforeEach, expect, test, vi } from 'vitest'; import AggregatorInterface from './AggregatorInterface'; @@ -9,6 +10,7 @@ vi.mock('../../services/api', () => ({ startAggregator: vi.fn(), stopAggregator: vi.fn(), uploadFile: vi.fn(), + deleteUploadedFile: vi.fn(), }, })); @@ -56,6 +58,7 @@ beforeEach(() => { total_rejections: 0, }); api.startAggregator.mockResolvedValue({ status: 'started' }); + api.deleteUploadedFile.mockResolvedValue({ status: 'deleted', deleted: true }); }); test('does not inherit Validator host provider when Assistant host is Auto', async () => { @@ -81,3 +84,64 @@ test('does not inherit Validator host provider when Assistant host is Auto', asy expect(payload.validator_openrouter_provider).toBe('AtlasCloud'); expect(payload.assistant_openrouter_provider).toBeNull(); }); + +test('keeps brainstorm start and creativity controls available in the prompt composer', async () => { + render( + <AggregatorInterface + config={{ ...baseConfig, creativityEmphasisBoostEnabled: false }} + setConfig={vi.fn()} + capabilities={{ lmStudioEnabled: true }} + developerModeEnabled={true} + /> + ); + + expect(await screen.findByLabelText('User Prompt *')).toHaveValue(baseConfig.userPrompt); + expect(screen.getByRole('button', { name: /start aggregator/i })).toBeEnabled(); + expect(screen.getByLabelText('Creativity Emphasis Boost')).toBeInTheDocument(); +}); + +test('shows uploaded lean files as removable attached context chips', async () => { + function Harness() { + const [config, setConfig] = React.useState(baseConfig); + return ( + <AggregatorInterface + config={config} + setConfig={setConfig} + capabilities={{ lmStudioEnabled: true }} + connectivityStatus={{ + skills: { + agent_conversation_memory: { enabled: true }, + }, + }} + /> + ); + } + + api.uploadFile.mockResolvedValue({ status: 'uploaded', filename: 'helper.lean', path: 'helper.lean' }); + + render(<Harness />); + + const uploadInput = await screen.findByLabelText('Attach Context Files (optional)'); + const leanFile = new File(['theorem helper : True := by trivial'], 'helper.lean', { + type: 'text/x-lean', + }); + + fireEvent.change(uploadInput, { target: { files: [leanFile] } }); + + expect(await screen.findByText('helper.lean')).toBeInTheDocument(); + fireEvent.click(screen.getByRole('button', { name: 'Remove helper.lean' })); + await waitFor(() => { + expect(api.deleteUploadedFile).toHaveBeenCalledWith('helper.lean'); + }); + expect(screen.queryByText('helper.lean')).not.toBeInTheDocument(); + + fireEvent.change(uploadInput, { target: { files: [leanFile] } }); + expect(await screen.findByText('helper.lean')).toBeInTheDocument(); + + fireEvent.click(screen.getByRole('button', { name: /start aggregator/i })); + + await waitFor(() => { + expect(api.startAggregator).toHaveBeenCalled(); + }); + expect(api.startAggregator.mock.calls[0][0].uploaded_files).toEqual(['helper.lean']); +}); diff --git a/frontend/src/components/aggregator/AggregatorLogs.jsx b/frontend/src/components/aggregator/AggregatorLogs.jsx index 808f37d..b55177a 100644 --- a/frontend/src/components/aggregator/AggregatorLogs.jsx +++ b/frontend/src/components/aggregator/AggregatorLogs.jsx @@ -6,6 +6,7 @@ import { MANUAL_AGGREGATOR_PROOF_SOURCE_ID } from '../../hooks/useProofCheckRunt import { formatContextOverflowActivityMessage, formatAssistantProofPackEventMessage, + formatSolutionPathEventMessage, buildRejectionFeedbackNoticeActivity, getActivityClass, getActivityIcon, @@ -36,8 +37,28 @@ const ASSISTANT_MEMORY_EVENTS = [ 'assistant_proof_pack_updated', 'assistant_proof_pack_failed', ]; +const SOLUTION_PATH_EVENTS = [ + 'solution_path_activated', + 'solution_path_proposal_queued', + 'solution_path_proposal_reviewing', + 'solution_path_updated', + 'solution_path_proposal_rejected', + 'solution_path_proposal_retry_queued', + 'solution_path_proposal_user_repair_required', + 'solution_path_proposal_resumed', +]; const HIDDEN_AGGREGATOR_ACTIVITY_EVENTS = new Set(['new_submission']); +export const shouldIncludeAggregatorProofContextOverflow = (data = {}) => ( + data.source_type === 'brainstorm' + && data.source_id === MANUAL_AGGREGATOR_PROOF_SOURCE_ID +); + +export const shouldIncludeAggregatorSolutionPathEvent = (data = {}) => { + const workflowMode = String(data.workflow_mode || data.mode || '').toLowerCase(); + return !workflowMode || workflowMode === 'aggregator'; +}; + const normalizeAggregatorEventName = (eventName = '') => { switch (eventName) { case 'accept': @@ -170,6 +191,10 @@ const countLatestRejectionStreak = (events) => { return count; }; +export const formatAggregatorPersistedOverflowMessage = (event = {}) => ( + formatContextOverflowActivityMessage(event.metadata || {}) +); + export default function AggregatorLogs() { const [events, setEvents] = useState([]); const [status, setStatus] = useState(null); @@ -197,9 +222,16 @@ export default function AggregatorLogs() { websocket.on('cleanup_review_complete', handleCleanupComplete), websocket.on('cleanup_review_error', handleCleanupError), websocket.on('context_overflow_error', handleContextOverflow), + websocket.on('proof_context_overflow', handleProofContextOverflow), websocket.on('hung_connection_alert', handleHungConnectionAlert), websocket.on('assistant_proof_pack_updated', (data) => handleAssistantProofPackEvent('assistant_proof_pack_updated', data)), websocket.on('assistant_proof_pack_failed', (data) => handleAssistantProofPackEvent('assistant_proof_pack_failed', data)), + ...SOLUTION_PATH_EVENTS.map((eventName) => ( + websocket.on(eventName, (data = {}) => { + if (!shouldIncludeAggregatorSolutionPathEvent(data)) return; + addEvent(eventName, formatSolutionPathEventMessage(eventName, data), data); + }) + )), ...MANUAL_PROOF_EVENTS.map((eventName) => ( websocket.on(eventName, (data) => handleManualProofEvent(eventName, data)) )), @@ -254,6 +286,12 @@ export default function AggregatorLogs() { message: formatAssistantProofPackEventMessage(event.type, event.data || {}), }; } + if (event.type === 'proof_context_overflow') { + return { + ...event, + message: formatContextOverflowActivityMessage(event.data || {}), + }; + } if (!MANUAL_PROOF_EVENTS.includes(event.type)) { return event; } @@ -268,6 +306,22 @@ export default function AggregatorLogs() { } }; + const formatPersistedEventMessage = (event = {}) => { + switch (event.type) { + case 'submission_accepted': + return `✓ ${event.message}`; + case 'submission_rejected': + return `✗ ${event.message}`; + case 'proof_attempt_failed': + case 'proof_attempts_exhausted': + return formatProofEvent(event.type, event.metadata || {}); + case 'context_overflow_error': + return formatAggregatorPersistedOverflowMessage(event); + default: + return event.message || event.type || 'Aggregator event'; + } + }; + const fetchBackendPersistedEvents = async () => { try { const response = await fetch('/api/aggregator/events'); @@ -294,20 +348,6 @@ export default function AggregatorLogs() { return []; }; - const formatPersistedEventMessage = (event = {}) => { - switch (event.type) { - case 'submission_accepted': - return `✓ ${event.message}`; - case 'submission_rejected': - return `✗ ${event.message}`; - case 'proof_attempt_failed': - case 'proof_attempts_exhausted': - return formatProofEvent(event.type, event.metadata || {}); - default: - return event.message || event.type || 'Aggregator event'; - } - }; - const loadInitialEvents = async () => { const [manualEvents, backendEvents] = await Promise.all([ Promise.resolve(readStoredManualEvents()), @@ -327,7 +367,10 @@ export default function AggregatorLogs() { const persistManualEvents = (nextEvents) => { try { const manualEvents = nextEvents.filter((event) => ( - MANUAL_PROOF_EVENTS.includes(event.type) || ASSISTANT_MEMORY_EVENTS.includes(event.type) + MANUAL_PROOF_EVENTS.includes(event.type) + || ASSISTANT_MEMORY_EVENTS.includes(event.type) + || SOLUTION_PATH_EVENTS.includes(event.type) + || event.type === 'proof_context_overflow' )); localStorage.setItem( AGGREGATOR_LIVE_ACTIVITY_STORAGE_KEY, @@ -406,6 +449,13 @@ export default function AggregatorLogs() { addEvent('context_overflow_error', formatContextOverflowActivityMessage(data), data); }; + const handleProofContextOverflow = (data = {}) => { + if (!shouldIncludeAggregatorProofContextOverflow(data)) { + return; + } + addEvent('proof_context_overflow', formatContextOverflowActivityMessage(data), data); + }; + const handleAssistantProofPackEvent = (eventName, data = {}) => { const workflowMode = String(data.workflow_mode || ''); const sourceId = String(data.source_id || ''); @@ -429,14 +479,23 @@ export default function AggregatorLogs() { addEvent(eventName, formatProofEvent(eventName, data), data); }; + const proofRoundPrefix = (data = {}) => { + const round = Number(data.proof_round_index || 0); + const maxRounds = Number(data.proof_max_rounds || 0); + return round > 0 && maxRounds > 1 ? `Proof round ${round}/${maxRounds} discovery` : 'Proof discovery'; + }; + const formatProofEvent = (eventName, data = {}) => { switch (eventName) { case 'proof_check_started': return 'Proof check started for the manual Aggregator database'; case 'proof_check_no_candidates': - return 'No formal theorem candidates found in the manual Aggregator database'; - case 'proof_check_candidates_found': - return `Proof candidates found: ${data.count || 0}`; + return `${proofRoundPrefix(data)} found 0 proof candidates; no proofs will be attempted`; + case 'proof_check_candidates_found': { + const count = Number(data.count || 0); + const subject = count === 1 ? 'proof candidate' : 'proof candidates'; + return `${proofRoundPrefix(data)} found ${count} ${subject}; ${count} will be attempted`; + } case 'proof_attempt_started': return `Lean proof attempt started: ${proofTargetLabel(data)}`; case 'proof_lean_accepted': diff --git a/frontend/src/components/aggregator/AggregatorSettings.jsx b/frontend/src/components/aggregator/AggregatorSettings.jsx index 868de65..96a1060 100644 --- a/frontend/src/components/aggregator/AggregatorSettings.jsx +++ b/frontend/src/components/aggregator/AggregatorSettings.jsx @@ -30,6 +30,7 @@ import { } from '../../utils/oauthProviders'; import HelpTooltip from '../HelpTooltip'; import HighlightedModelsSidebar from '../HighlightedModelsSidebar'; +import OpenRouterFreeModelsControl from '../OpenRouterFreeModelsControl'; import ProofStrengthBadge from '../ProofStrengthBadge'; import RawSettingsEditor from '../RawSettingsEditor'; import '../autonomous/AutonomousResearch.css'; @@ -292,8 +293,8 @@ export default function AggregatorSettings({ const [sakanaFuguModelError, setSakanaFuguModelError] = useState(''); const [loadingOpenRouter, setLoadingOpenRouter] = useState(false); const [freeOnly, setFreeOnly] = useState(false); - const [freeModelLooping, setFreeModelLooping] = useState(true); - const [freeModelAutoSelector, setFreeModelAutoSelector] = useState(true); + const [freeModelLooping, setFreeModelLooping] = useState(false); + const [freeModelAutoSelector, setFreeModelAutoSelector] = useState(false); const [isLoaded, setIsLoaded] = useState(false); const [editRawSettings, setEditRawSettings] = useState(false); const [rawSettingsText, setRawSettingsText] = useState(''); @@ -350,8 +351,8 @@ export default function AggregatorSettings({ } try { const freeModelSettings = await openRouterAPI.getFreeModelSettings(); - setFreeModelLooping(freeModelSettings.looping_enabled ?? true); - setFreeModelAutoSelector(freeModelSettings.auto_selector_enabled ?? true); + setFreeModelLooping(freeModelSettings.looping_enabled ?? false); + setFreeModelAutoSelector(freeModelSettings.auto_selector_enabled ?? false); } catch (error) { console.error('Failed to load free model settings:', error); } @@ -411,9 +412,10 @@ export default function AggregatorSettings({ freeModelAutoSelector, modelProviders, creativityEmphasisBoostEnabled: config.creativityEmphasisBoostEnabled, + uploadedFiles: config.uploadedFiles || [], }; localStorage.setItem('aggregator_settings', JSON.stringify(settings)); - }, [isLoaded, config.userPrompt, config.validatorModel, config.validatorContextSize, config.assistantModel, config.assistantProvider, config.assistantOpenrouterProvider, config.assistantOpenrouterReasoningEffort, config.assistantLmStudioFallback, config.assistantContextSize, config.assistantMaxOutput, config.assistantSuperchargeEnabled, numSubmitters, submitterConfigs, validatorProvider, validatorOpenrouterProvider, validatorOpenrouterReasoningEffort, validatorLmStudioFallback, validatorSuperchargeEnabled, validatorMaxOutput, freeOnly, freeModelLooping, freeModelAutoSelector, modelProviders, config.creativityEmphasisBoostEnabled]); + }, [isLoaded, config.userPrompt, config.validatorModel, config.validatorContextSize, config.assistantModel, config.assistantProvider, config.assistantOpenrouterProvider, config.assistantOpenrouterReasoningEffort, config.assistantLmStudioFallback, config.assistantContextSize, config.assistantMaxOutput, config.assistantSuperchargeEnabled, numSubmitters, submitterConfigs, validatorProvider, validatorOpenrouterProvider, validatorOpenrouterReasoningEffort, validatorLmStudioFallback, validatorSuperchargeEnabled, validatorMaxOutput, freeOnly, freeModelLooping, freeModelAutoSelector, modelProviders, config.creativityEmphasisBoostEnabled, config.uploadedFiles]); useEffect(() => { if (lmStudioEnabled) { @@ -1081,11 +1083,11 @@ export default function AggregatorSettings({ setValidatorSuperchargeEnabled(nextValidatorSuperchargeEnabled); setValidatorMaxOutput(nextValidatorMaxOutput); setFreeOnly(rawSettings.freeOnly ?? false); - setFreeModelLooping(rawSettings.freeModelLooping ?? true); - setFreeModelAutoSelector(rawSettings.freeModelAutoSelector ?? true); + setFreeModelLooping(rawSettings.freeModelLooping ?? false); + setFreeModelAutoSelector(rawSettings.freeModelAutoSelector ?? false); setModelProviders(nextModelProviders); openRouterAPI - .setFreeModelSettings(rawSettings.freeModelLooping ?? true, rawSettings.freeModelAutoSelector ?? true) + .setFreeModelSettings(rawSettings.freeModelLooping ?? false, rawSettings.freeModelAutoSelector ?? false) .catch(() => {}); const hasRawUserPrompt = Object.prototype.hasOwnProperty.call(rawSettings, 'userPrompt'); @@ -1120,8 +1122,8 @@ export default function AggregatorSettings({ freeOnly: rawSettings.freeOnly ?? false, validatorOpenrouterReasoningEffort: nextValidatorOpenrouterReasoningEffort, validatorSuperchargeEnabled: nextValidatorSuperchargeEnabled, - freeModelLooping: rawSettings.freeModelLooping ?? true, - freeModelAutoSelector: rawSettings.freeModelAutoSelector ?? true, + freeModelLooping: rawSettings.freeModelLooping ?? false, + freeModelAutoSelector: rawSettings.freeModelAutoSelector ?? false, modelProviders: nextModelProviders, })); } @@ -1209,17 +1211,15 @@ export default function AggregatorSettings({ <button onClick={() => fetchOpenRouterModels(freeOnly)} className="secondary" disabled={loadingOpenRouter}> {loadingOpenRouter ? 'Loading...' : 'Refresh OpenRouter Models'} </button> - <label className="settings-checkbox-label model-refresh-controls__toggle"> - <input - type="checkbox" - checked={freeOnly} - onChange={(e) => setFreeOnly(e.target.checked)} - /> - Free models only - </label> + <OpenRouterFreeModelsControl + checked={freeOnly} + onChange={setFreeOnly} + /> </> )} - {developerModeEnabled ? ( + {developerModeEnabled && ( + <> + {hasOpenRouterKey && <span className="model-refresh-controls__divider" aria-hidden="true" />} <label className="settings-checkbox-label model-refresh-controls__toggle"> <input type="checkbox" @@ -1228,10 +1228,7 @@ export default function AggregatorSettings({ /> Edit Raw </label> - ) : ( - <span className="settings-developer-mode-hint"> - Developer mode: press Shift + Z + X to toggle raw JSON settings. - </span> + </> )} </div> @@ -1484,7 +1481,7 @@ export default function AggregatorSettings({ <div className="settings-group"> <h4>Assistant</h4> <p className="settings-info"> - Runs in parallel during brainstorming and proof work to retrieve up to 7 relevant memory supports from Session History Memory and SyntheticLib4 when enabled. Validators and critique phases never receive Assistant context. + Runs in parallel during brainstorming and proof work to retrieve up to 7 relevant verified proof-memory supports from Session History Memory and SyntheticLib4 when enabled. Validators and critique phases never receive Assistant context. </p> <div className={`submitter-config-section${(config.assistantProvider || validatorProvider) === 'openrouter' ? ' role-config-card--openrouter-orange' : ''}`} diff --git a/frontend/src/components/aggregator/AggregatorSettings.test.jsx b/frontend/src/components/aggregator/AggregatorSettings.test.jsx index d37b148..a1d2220 100644 --- a/frontend/src/components/aggregator/AggregatorSettings.test.jsx +++ b/frontend/src/components/aggregator/AggregatorSettings.test.jsx @@ -94,8 +94,8 @@ beforeEach(() => { localStorage.clear(); openRouterAPI.getApiKeyStatus.mockResolvedValue({ has_key: true }); openRouterAPI.getFreeModelSettings.mockResolvedValue({ - looping_enabled: true, - auto_selector_enabled: true, + looping_enabled: false, + auto_selector_enabled: false, }); openRouterAPI.getModels.mockResolvedValue({ models: [ diff --git a/frontend/src/components/autonomous/AutonomousResearch.css b/frontend/src/components/autonomous/AutonomousResearch.css index dca1376..6e7821d 100644 --- a/frontend/src/components/autonomous/AutonomousResearch.css +++ b/frontend/src/components/autonomous/AutonomousResearch.css @@ -421,6 +421,23 @@ gap: 0.5rem; } +.prompt-composer-section { + gap: 0.65rem; +} + +.prompt-composer { + display: flex; + flex-direction: column; + gap: 0.65rem; + padding: 0.9rem; + background: + linear-gradient(180deg, rgba(255, 255, 255, 0.035), rgba(255, 255, 255, 0.01)), + var(--bg-secondary, #1e1e1e); + border: 1px solid var(--border-color, #333); + border-radius: 18px; + box-shadow: 0 12px 32px rgba(0, 0, 0, 0.24); +} + .research-prompt-section label { font-weight: 600; color: var(--text-primary, #e0e0e0); @@ -438,17 +455,87 @@ font-family: inherit; } +.prompt-composer textarea { + min-height: 140px; + border: none; + border-radius: 12px; + padding: 0.85rem 0.9rem; + background: rgba(8, 8, 12, 0.48); + resize: vertical; +} + .research-prompt-section textarea:focus { outline: none; border-color: #18cc17; box-shadow: 0 0 0 2px rgba(24, 204, 23, 0.2); } +.prompt-composer textarea:focus { + border-color: transparent; + box-shadow: inset 0 0 0 1px rgba(24, 204, 23, 0.72), 0 0 0 3px rgba(24, 204, 23, 0.16); +} + .research-prompt-section textarea:disabled { opacity: 0.7; cursor: not-allowed; } +.prompt-composer-actions { + display: flex; + align-items: flex-start; + justify-content: space-between; + gap: 0.75rem; +} + +.prompt-composer-actions .text-file-uploader { + flex: 1 1 240px; + min-width: 180px; + margin: 0; +} + +.prompt-composer-actions .text-upload-btn { + min-height: 42px; + border-radius: 999px; +} + +.prompt-composer-primary { + flex: 0 0 auto; + min-height: 42px; + border-radius: 999px; + padding-inline: 1.35rem; + white-space: nowrap; +} + +.prompt-composer-running-actions { + display: inline-flex; + flex: 0 0 auto; + align-items: center; + justify-content: flex-end; + gap: 0.65rem; + flex-wrap: wrap; +} + +.prompt-composer-options { + display: flex; + align-items: center; + justify-content: flex-end; + gap: 0.65rem; + flex-wrap: wrap; + width: 100%; +} + +.prompt-composer-option { + margin-bottom: 0; + width: auto; +} + +.prompt-composer-options .btn-clear, +.prompt-composer-options .btn-cancel { + min-height: 34px; + padding: 0.42rem 0.85rem; + border-radius: 999px; +} + /* Status Section */ .status-section { background: var(--bg-secondary, #1e1e1e); @@ -1141,19 +1228,18 @@ } /* Settings - Layout with Left Sidebar */ -.autonomous-settings-layout { +.autonomous-settings-layout, +.settings-with-model-sidebar { display: flex; gap: 0; - height: 100%; width: 100%; min-height: 100vh; + padding-right: var(--workflow-panel-safe-area); + box-sizing: border-box; } -.settings-with-model-sidebar { - display: flex; - gap: 0; - width: 100%; - min-height: 100vh; +.autonomous-settings-layout { + height: 100%; } .settings-with-model-sidebar__main { @@ -1161,23 +1247,13 @@ min-width: 0; padding: 1.5rem; overflow-y: auto; - width: 100%; - max-width: calc(100vw - 322px - 3rem); + width: auto; + max-width: none; box-sizing: border-box; position: relative; z-index: 10; } -.app.workflow-panel-expanded .settings-with-model-sidebar__main { - max-width: calc(100vw - 322px - 3rem - 320px); - padding-right: 1.5rem; -} - -.app.workflow-panel-collapsed .settings-with-model-sidebar__main { - max-width: calc(100vw - 322px - 3rem - 50px); - padding-right: 1.5rem; -} - .settings-left-sidebar { width: 322px; background: linear-gradient(180deg, rgba(15, 19, 16, 0.98) 0%, rgba(20, 29, 22, 0.96) 100%); @@ -1439,32 +1515,19 @@ /* Main settings content area */ .autonomous-settings { flex: 1; + min-width: 0; padding: 1.5rem; display: flex; flex-direction: column; gap: 1.5rem; overflow-y: auto; - width: 100%; - /* Default: accounts for left sidebar (322px) + horizontal padding (3rem). */ - max-width: calc(100vw - 322px - 3rem); + width: auto; + max-width: none; box-sizing: border-box; position: relative; z-index: 1; } -/* When the WorkflowPanel sidebar is expanded (320px) or collapsed (50px), - slide settings content leftward so dropdowns/arrows don't get clipped - by the fixed-position panel overlay. */ -.app.workflow-panel-expanded .autonomous-settings { - max-width: calc(100vw - 322px - 3rem - 320px); - padding-right: 1.5rem; -} - -.app.workflow-panel-collapsed .autonomous-settings { - max-width: calc(100vw - 322px - 3rem - 50px); - padding-right: 1.5rem; -} - .settings-group { background: var(--bg-secondary, #1e1e1e); border: 1px solid #2a2a2a; @@ -3092,22 +3155,44 @@ /* Responsive */ /* ============================================================ */ -@media (max-width: 768px) { - .autonomous-settings-layout, - .settings-with-model-sidebar { +@media (min-width: 769px) and (max-width: 1119px) { + .app.workflow-panel-expanded .autonomous-settings-layout, + .app.workflow-panel-expanded .settings-with-model-sidebar { flex-direction: column; } - - .settings-left-sidebar { + + .app.workflow-panel-expanded .settings-left-sidebar { width: 100%; border-right: none; border-bottom: 2px solid rgba(76, 175, 80, 0.16); max-height: 300px; } - +} + +@media (min-width: 769px) and (max-width: 829px) { + .app.workflow-panel-collapsed .autonomous-settings-layout, + .app.workflow-panel-collapsed .settings-with-model-sidebar { + flex-direction: column; + } + + .app.workflow-panel-collapsed .settings-left-sidebar { + width: 100%; + border-right: none; + border-bottom: 2px solid rgba(76, 175, 80, 0.16); + max-height: 300px; + } +} + +@media (max-width: 768px) { + .autonomous-settings-layout, + .settings-with-model-sidebar { + padding-right: 0; + } + .autonomous-settings, .settings-with-model-sidebar__main { padding: 1rem; + width: 100%; max-width: 100%; } @@ -3141,6 +3226,37 @@ .delete-confirm-inline { flex-wrap: wrap; } + + .prompt-composer { + padding: 0.75rem; + border-radius: 14px; + } + + .prompt-composer-actions { + flex-direction: column; + align-items: stretch; + } + + .prompt-composer-actions .text-file-uploader, + .prompt-composer-primary, + .prompt-composer-running-actions, + .prompt-composer-running-actions .btn-stop { + width: 100%; + } + + .prompt-composer-running-actions { + justify-content: stretch; + } + + .prompt-composer-options { + justify-content: flex-start; + } + + .prompt-composer-options .allowed-outputs-row { + width: 100%; + justify-content: flex-start; + overflow-x: auto; + } .tier3-dialog { padding: 1.5rem; diff --git a/frontend/src/components/autonomous/AutonomousResearchInterface.jsx b/frontend/src/components/autonomous/AutonomousResearchInterface.jsx index 8c20e3e..d81da40 100644 --- a/frontend/src/components/autonomous/AutonomousResearchInterface.jsx +++ b/frontend/src/components/autonomous/AutonomousResearchInterface.jsx @@ -29,6 +29,7 @@ const AutonomousResearchInterface = ({ status, activity, onStart, + onStartingChange, onStop, onClear, config, @@ -149,11 +150,13 @@ const AutonomousResearchInterface = ({ } }, [isRunning, isStopping]); - const handleTextFileLoaded = (content) => { - // Append to existing prompt with separator - const separator = researchPrompt.trim() ? '\n\n' : ''; - const newPrompt = researchPrompt + separator + content; - setResearchPrompt(newPrompt); + const handleTextFileLoaded = (content, filename = 'uploaded-file') => { + const isLeanFile = filename.toLowerCase().endsWith('.lean'); + const labeledContext = `[UPLOADED ${isLeanFile ? 'LEAN FILE' : 'TEXT FILE'}: ${filename}]\n${content}`; + setResearchPrompt(prevPrompt => { + const separator = prevPrompt.trim() ? '\n\n' : ''; + return prevPrompt + separator + labeledContext; + }); }; const handleStart = async () => { @@ -173,12 +176,14 @@ const AutonomousResearchInterface = ({ return; } setStartRequested(true); + onStartingChange?.(true); const proofOnlyRequested = mathematicalProofsAllowed && !researchPapersAllowed; const shouldSyncProofRuntime = mathematicalProofsAllowed && !capabilities?.genericMode; if (proofOnlyRequested || shouldSyncProofRuntime) { const enabled = await updateProofRuntimeSetting(true); if (!enabled) { setStartRequested(false); + onStartingChange?.(false); return; } } @@ -190,6 +195,8 @@ const AutonomousResearchInterface = ({ } catch (error) { setStartRequested(false); throw error; + } finally { + onStartingChange?.(false); } }; @@ -212,7 +219,7 @@ const AutonomousResearchInterface = ({ lean4_lsp_idle_timeout: status.lean4_lsp_idle_timeout ?? 600, max_parallel_candidates: status.proof_max_parallel_candidates ?? 6, smt_enabled: Boolean(status.smt_enabled), - smt_timeout: status.smt_timeout ?? 30, + smt_timeout: status.smt_timeout ?? 300, }); if (enabled) { const leanVersion = String(updatedStatus.lean4_version || updatedStatus.lean_version || '').trim(); @@ -349,11 +356,33 @@ const AutonomousResearchInterface = ({ {/* Header */} <div className="autonomous-header"> <h2>Autonomous Research</h2> - <div className="autonomous-controls-stack"> - <div className="autonomous-controls"> + </div> + + {/* Research Prompt Input */} + <div className="research-prompt-section prompt-composer-section"> + <div className="prompt-composer"> + <label htmlFor="research-prompt">Research Goal</label> + <textarea + id="research-prompt" + value={researchPrompt} + onChange={(e) => setResearchPrompt(e.target.value)} + placeholder="Enter a high-level S.T.E.M. research or solution objective (e.g., 'Advance desalination technology,' 'Design a resilient distributed protocol,' or 'Solve a mathematical conjecture')" + disabled={controlsLocked} + rows={3} + /> + <div className="prompt-composer-actions"> + <TextFileUploader + onFileLoaded={handleTextFileLoaded} + disabled={controlsLocked} + maxSizeMB={5} + showCharCount={true} + confirmIfNotEmpty={true} + existingPromptLength={researchPrompt.length} + /> {!showStopControl ? ( <button - className="btn-start" + type="button" + className="btn-start prompt-composer-primary" onClick={handleStart} disabled={ !config?.submitter_configs?.some(s => s.modelId) || @@ -363,7 +392,7 @@ const AutonomousResearchInterface = ({ Start Research </button> ) : ( - <> + <div className="prompt-composer-running-actions"> {showRuntimeIndicator && ( <span className="runtime-indicator" @@ -375,41 +404,19 @@ const AutonomousResearchInterface = ({ <span className="runtime-indicator-label">{isStopping ? 'Stopping' : 'Running'}</span> </span> )} - <button className="btn-stop" onClick={onStop} disabled={isStopping}> - {isStopping ? 'Stopping...' : 'Stop Research'} + <button + type="button" + className="btn-stop" + onClick={onStop} + disabled={isStopping || (startRequested && !isRunning)} + > + {isStopping ? 'Stopping...' : (startRequested && !isRunning ? 'Starting...' : 'Stop Research')} </button> - </> - )} - {developerModeEnabled && ( - <label className="settings-checkbox-label"> - <input - type="checkbox" - checked={Boolean(config?.creativity_emphasis_boost_enabled)} - onChange={(event) => onConfigChange?.({ - ...config, - creativity_emphasis_boost_enabled: event.target.checked - })} - disabled={controlsLocked} - /> - Creativity Emphasis Boost - </label> - )} - <button - className={`btn-clear ${showClearConfirm ? 'btn-confirm' : ''}`} - onClick={handleClear} - disabled={controlsLocked || isClearing} - > - {isClearing ? 'Clearing...' : (showClearConfirm ? 'Confirm Clear' : 'Clear All')} - </button> - {showClearConfirm && !isClearing && ( - <button - className="btn-cancel" - onClick={() => setShowClearConfirm(false)} - > - Cancel - </button> + </div> )} </div> + </div> + <div className="prompt-composer-options"> <div className="allowed-outputs-row" title="Allowed Outputs controls which products this workflow may generate. At least one output must remain enabled." @@ -440,30 +447,40 @@ const AutonomousResearchInterface = ({ <span className="allowed-output-text">Research Papers</span> </label> </div> + {developerModeEnabled && ( + <label className="settings-checkbox-label prompt-composer-option"> + <input + type="checkbox" + checked={Boolean(config?.creativity_emphasis_boost_enabled)} + onChange={(event) => onConfigChange?.({ + ...config, + creativity_emphasis_boost_enabled: event.target.checked + })} + disabled={controlsLocked} + /> + Creativity Emphasis Boost + </label> + )} + <button + type="button" + className={`btn-clear ${showClearConfirm ? 'btn-confirm' : ''}`} + onClick={handleClear} + disabled={controlsLocked || isClearing} + > + {isClearing ? 'Clearing...' : (showClearConfirm ? 'Confirm Reset' : 'Clear Research Run')} + </button> + {showClearConfirm && !isClearing && ( + <button + type="button" + className="btn-cancel" + onClick={() => setShowClearConfirm(false)} + > + Cancel + </button> + )} </div> </div> - {/* Research Prompt Input */} - <div className="research-prompt-section"> - <label htmlFor="research-prompt">Research Goal</label> - <textarea - id="research-prompt" - value={researchPrompt} - onChange={(e) => setResearchPrompt(e.target.value)} - placeholder="Enter your high level research goal on any topic that relates to S.T.E.M. mathematics, anything remotely related to mathematics (e.g., 'Advance desalination technology' or 'Solve physics unification')" - disabled={controlsLocked} - rows={3} - /> - <TextFileUploader - onFileLoaded={handleTextFileLoaded} - disabled={controlsLocked} - maxSizeMB={5} - showCharCount={true} - confirmIfNotEmpty={true} - existingPromptLength={researchPrompt.length} - /> - </div> - {/* Status Display */} <div className="status-section"> <div className="status-tier"> diff --git a/frontend/src/components/autonomous/AutonomousResearchInterface.test.jsx b/frontend/src/components/autonomous/AutonomousResearchInterface.test.jsx index 0c97293..f65dafc 100644 --- a/frontend/src/components/autonomous/AutonomousResearchInterface.test.jsx +++ b/frontend/src/components/autonomous/AutonomousResearchInterface.test.jsx @@ -57,6 +57,101 @@ test('hydrates autonomous prompt synchronously from session storage fallback', ( expect(screen.getByLabelText('Research Goal')).toHaveValue('cached autonomous prompt'); }); +test('keeps autonomous start and output controls available in the prompt composer', () => { + render( + <AutonomousResearchInterface + {...baseProps} + config={{ + ...baseProps.config, + submitter_configs: [{ modelId: 'openai/gpt-5.5' }], + }} + capabilities={{ genericMode: false }} + developerModeEnabled={true} + /> + ); + + expect(screen.getByLabelText('Research Goal')).toHaveAttribute( + 'placeholder', + expect.stringContaining('S.T.E.M. research or solution objective') + ); + expect(screen.getByRole('button', { name: 'Start Research' })).toBeEnabled(); + expect(screen.getByLabelText('Mathematical Proofs')).toBeInTheDocument(); + expect(screen.getByLabelText('Research Papers')).toBeInTheDocument(); + expect(screen.getByLabelText('Creativity Emphasis Boost')).toBeInTheDocument(); +}); + +test('shows the red stop control immediately but locks it until start is confirmed', async () => { + let resolveStart; + const onStart = vi.fn(() => new Promise((resolve) => { + resolveStart = resolve; + })); + sessionStorage.setItem('autonomous_research_prompt', 'research this safely'); + + const { rerender } = render( + <AutonomousResearchInterface + {...baseProps} + onStart={onStart} + config={{ + ...baseProps.config, + submitter_configs: [{ modelId: 'openai/gpt-5.5' }], + }} + /> + ); + + fireEvent.click(screen.getByRole('button', { name: 'Start Research' })); + + await waitFor(() => { + expect(onStart).toHaveBeenCalledTimes(1); + }); + expect(screen.getByRole('button', { name: 'Starting...' })).toBeDisabled(); + + rerender( + <AutonomousResearchInterface + {...baseProps} + isRunning={true} + onStart={onStart} + config={{ + ...baseProps.config, + submitter_configs: [{ modelId: 'openai/gpt-5.5' }], + }} + /> + ); + resolveStart(true); + + await waitFor(() => { + expect(screen.getByRole('button', { name: 'Stop Research' })).toBeEnabled(); + }); +}); + +test('reports pending start to the parent so settings can remain locked', async () => { + let resolveStart; + const onStart = vi.fn(() => new Promise((resolve) => { + resolveStart = resolve; + })); + const onStartingChange = vi.fn(); + sessionStorage.setItem('autonomous_research_prompt', 'research this safely'); + + render( + <AutonomousResearchInterface + {...baseProps} + onStart={onStart} + onStartingChange={onStartingChange} + config={{ + ...baseProps.config, + submitter_configs: [{ modelId: 'openai/gpt-5.5' }], + }} + /> + ); + + fireEvent.click(screen.getByRole('button', { name: 'Start Research' })); + expect(onStartingChange).toHaveBeenCalledWith(true); + + resolveStart(false); + await waitFor(() => { + expect(onStartingChange).toHaveBeenLastCalledWith(false); + }); +}); + test('keeps autonomous prompt when confirmed clear is canceled by parent handler', async () => { const onClear = vi.fn().mockResolvedValue(false); sessionStorage.setItem('autonomous_research_prompt', 'prompt should stay'); @@ -66,8 +161,8 @@ test('keeps autonomous prompt when confirmed clear is canceled by parent handler const promptInput = screen.getByLabelText('Research Goal'); expect(promptInput).toHaveValue('prompt should stay'); - fireEvent.click(screen.getByRole('button', { name: 'Clear All' })); - fireEvent.click(screen.getByRole('button', { name: 'Confirm Clear' })); + fireEvent.click(screen.getByRole('button', { name: 'Clear Research Run' })); + fireEvent.click(screen.getByRole('button', { name: 'Confirm Reset' })); await waitFor(() => { expect(onClear).toHaveBeenCalledTimes(1); @@ -85,8 +180,8 @@ test('clears autonomous prompt after confirmed clear succeeds', async () => { const promptInput = screen.getByLabelText('Research Goal'); expect(promptInput).toHaveValue('prompt should clear'); - fireEvent.click(screen.getByRole('button', { name: 'Clear All' })); - fireEvent.click(screen.getByRole('button', { name: 'Confirm Clear' })); + fireEvent.click(screen.getByRole('button', { name: 'Clear Research Run' })); + fireEvent.click(screen.getByRole('button', { name: 'Confirm Reset' })); await waitFor(() => { expect(promptInput).toHaveValue(''); diff --git a/frontend/src/components/autonomous/AutonomousResearchLogs.jsx b/frontend/src/components/autonomous/AutonomousResearchLogs.jsx index fef769b..31077ae 100644 --- a/frontend/src/components/autonomous/AutonomousResearchLogs.jsx +++ b/frontend/src/components/autonomous/AutonomousResearchLogs.jsx @@ -198,9 +198,13 @@ const AutonomousResearchLogs = ({ stats, events }) => { case 'proof_retry_started': return `Retrying ${data.count || 0} failed proof candidate(s) against paper ${data.source_id}`; case 'proof_check_no_candidates': - return `${proofRoundLabel() || 'Proof check'} found no formal theorem candidates in ${data.source_type} ${data.source_id}`; - case 'proof_check_candidates_found': - return `${proofRoundLabel() || 'Proof check'} candidates found: ${data.count || 0}`; + return `${proofRoundLabel() ? `${proofRoundLabel()} discovery` : 'Proof discovery'} found 0 proof candidates; no proofs will be attempted`; + case 'proof_check_candidates_found': { + const count = Number(data.count || 0); + const subject = count === 1 ? 'proof candidate' : 'proof candidates'; + const prefix = proofRoundLabel() ? `${proofRoundLabel()} discovery` : 'Proof discovery'; + return `${prefix} found ${count} ${subject}; ${count} will be attempted`; + } case 'proof_attempt_started': return `${proofName}, Attempt ${data.attempt || 1} started: ${proofTarget}`; case 'proof_attempt_failed': diff --git a/frontend/src/components/autonomous/AutonomousResearchSettings.jsx b/frontend/src/components/autonomous/AutonomousResearchSettings.jsx index 803baa5..5f59948 100644 --- a/frontend/src/components/autonomous/AutonomousResearchSettings.jsx +++ b/frontend/src/components/autonomous/AutonomousResearchSettings.jsx @@ -46,6 +46,7 @@ import { } from '../../utils/autonomousProfiles'; import HelpTooltip from '../HelpTooltip'; import HighlightedModelsSidebar from '../HighlightedModelsSidebar'; +import OpenRouterFreeModelsControl from '../OpenRouterFreeModelsControl'; import ProofStrengthBadge from '../ProofStrengthBadge'; import RawSettingsEditor from '../RawSettingsEditor'; import { readBooleanStorage } from '../../utils/safeStorage'; @@ -442,8 +443,8 @@ const AutonomousResearchSettings = ({ const [sakanaFuguModelError, setSakanaFuguModelError] = useState(''); const [loadingOpenRouter, setLoadingOpenRouter] = useState(false); const [freeOnly, setFreeOnly] = useState(false); - const [freeModelLooping, setFreeModelLooping] = useState(true); - const [freeModelAutoSelector, setFreeModelAutoSelector] = useState(true); + const [freeModelLooping, setFreeModelLooping] = useState(false); + const [freeModelAutoSelector, setFreeModelAutoSelector] = useState(false); const [tier3Enabled, setTier3Enabled] = useState(false); const [isLoadedFromStorage, setIsLoadedFromStorage] = useState(false); @@ -465,7 +466,7 @@ const AutonomousResearchSettings = ({ const [proofSettingsLspIdleTimeout, setProofSettingsLspIdleTimeout] = useState('600'); const [proofSettingsMaxParallelCandidates, setProofSettingsMaxParallelCandidates] = useState('6'); const [proofSettingsSmtEnabled, setProofSettingsSmtEnabled] = useState(false); - const [proofSettingsSmtTimeout, setProofSettingsSmtTimeout] = useState('30'); + const [proofSettingsSmtTimeout, setProofSettingsSmtTimeout] = useState('300'); const [savingProofSettings, setSavingProofSettings] = useState(false); const [proofSettingsMessage, setProofSettingsMessage] = useState(''); @@ -659,8 +660,8 @@ const AutonomousResearchSettings = ({ try { const freeModelSettings = await openRouterAPI.getFreeModelSettings(); - setFreeModelLooping(freeModelSettings.looping_enabled ?? true); - setFreeModelAutoSelector(freeModelSettings.auto_selector_enabled ?? true); + setFreeModelLooping(freeModelSettings.looping_enabled ?? false); + setFreeModelAutoSelector(freeModelSettings.auto_selector_enabled ?? false); } catch (err) { console.error('Failed to load free model settings:', err); } @@ -765,7 +766,7 @@ const AutonomousResearchSettings = ({ setProofSettingsLspIdleTimeout(String(status.lean4_lsp_idle_timeout ?? 600)); setProofSettingsMaxParallelCandidates(String(status.proof_max_parallel_candidates ?? 6)); setProofSettingsSmtEnabled(Boolean(status.smt_enabled)); - setProofSettingsSmtTimeout(String(status.smt_timeout ?? 30)); + setProofSettingsSmtTimeout(String(status.smt_timeout ?? 300)); } catch (err) { console.error('Failed to load Lean 4 proof status:', err); } @@ -1526,7 +1527,7 @@ Be honest and constructive. Identify both strengths and weaknesses.`; ? Math.max(0, parsedMaxParallelCandidates) : 6; const parsedSmtTimeout = parseInt(proofSettingsSmtTimeout, 10); - const smtTimeout = Number.isFinite(parsedSmtTimeout) ? parsedSmtTimeout : 30; + const smtTimeout = Number.isFinite(parsedSmtTimeout) ? parsedSmtTimeout : 300; try { setSavingProofSettings(true); @@ -1950,22 +1951,16 @@ Be honest and constructive. Identify both strengths and weaknesses.`; > 🔗 OpenRouter Model List </button> - <label - className="settings-checkbox-label model-refresh-controls__toggle" - style={{ cursor: isRunning ? 'not-allowed' : 'pointer' }} - > - <input - type="checkbox" - checked={freeOnly} - onChange={(e) => setFreeOnly(e.target.checked)} - disabled={isRunning} - style={{ marginRight: '0.5rem' }} - /> - Free models only - </label> + <OpenRouterFreeModelsControl + checked={freeOnly} + disabled={isRunning} + onChange={setFreeOnly} + /> </> )} - {developerModeEnabled ? ( + {developerModeEnabled && ( + <> + {hasOpenRouterKey && <span className="model-refresh-controls__divider" aria-hidden="true" />} <label className="settings-checkbox-label model-refresh-controls__toggle" style={{ cursor: isRunning ? 'not-allowed' : 'pointer' }} @@ -1978,10 +1973,7 @@ Be honest and constructive. Identify both strengths and weaknesses.`; /> Edit Raw </label> - ) : ( - <span className="settings-developer-mode-hint"> - Developer mode: press Shift + Z + X to toggle raw JSON settings. - </span> + </> )} </div> @@ -2157,7 +2149,7 @@ Be honest and constructive. Identify both strengths and weaknesses.`; <RoleConfig title="Assistant" - hint="Runs in parallel during brainstorming, writing, proof work, and final-answer work to retrieve up to 7 relevant memory supports from Session History Memory and SyntheticLib4 when enabled. Validators and critique phases do not receive Assistant context." + hint="Runs in parallel during brainstorming, writing, proof work, and final-answer work to retrieve up to 7 relevant verified proof-memory supports from Session History Memory and SyntheticLib4 when enabled. Validators and critique phases do not receive Assistant context." rolePrefix="assistant" localConfig={localConfig} handleProviderChange={handleProviderChange} diff --git a/frontend/src/components/autonomous/AutonomousResearchSettings.test.jsx b/frontend/src/components/autonomous/AutonomousResearchSettings.test.jsx index 888826f..b446db3 100644 --- a/frontend/src/components/autonomous/AutonomousResearchSettings.test.jsx +++ b/frontend/src/components/autonomous/AutonomousResearchSettings.test.jsx @@ -82,8 +82,8 @@ beforeEach(() => { localStorage.clear(); openRouterAPI.getApiKeyStatus.mockResolvedValue({ has_key: true }); openRouterAPI.getFreeModelSettings.mockResolvedValue({ - looping_enabled: true, - auto_selector_enabled: true, + looping_enabled: false, + auto_selector_enabled: false, }); openRouterAPI.getModels.mockResolvedValue({ models: [ @@ -103,7 +103,7 @@ beforeEach(() => { lean4_lsp_enabled: false, proof_max_parallel_candidates: 6, smt_enabled: false, - smt_timeout: 30, + smt_timeout: 300, }); autonomousAPI.getDefaultCritiquePrompt.mockResolvedValue({ data: { diff --git a/frontend/src/components/autonomous/FinalAnswerLibrary.jsx b/frontend/src/components/autonomous/FinalAnswerLibrary.jsx index 99ca130..eb0dec5 100644 --- a/frontend/src/components/autonomous/FinalAnswerLibrary.jsx +++ b/frontend/src/components/autonomous/FinalAnswerLibrary.jsx @@ -38,7 +38,7 @@ function FinalAnswerLibrary({ capabilities }) { const [expandedContent, setExpandedContent] = useState(null); const [searchTerm, setSearchTerm] = useState(''); const [filterFormat, setFilterFormat] = useState('all'); // 'all', 'short_form', 'long_form' - const [showLatex, setShowLatex] = useState(false); // Raw text by default for performance with large docs + const [showLatex, setShowLatex] = useState(true); const [downloadingPDF, setDownloadingPDF] = useState(null); // Track which answer is generating PDF const [expandedPrunedRuns, setExpandedPrunedRuns] = useState({}); const [expandedPrunedPaperId, setExpandedPrunedPaperId] = useState(null); @@ -673,7 +673,7 @@ function FinalAnswerLibrary({ capabilities }) { : expandedPrunedContent.content || '' } showToggle={true} - defaultRaw={true} + defaultRaw={false} /> </div> )} diff --git a/frontend/src/components/autonomous/FinalAnswerView.jsx b/frontend/src/components/autonomous/FinalAnswerView.jsx index 225a4d2..6b4d900 100644 --- a/frontend/src/components/autonomous/FinalAnswerView.jsx +++ b/frontend/src/components/autonomous/FinalAnswerView.jsx @@ -28,7 +28,7 @@ const FinalAnswerView = ({ api, isRunning, status, capabilities }) => { const [activeSection, setActiveSection] = useState('overview'); const [isRegenerating, setIsRegenerating] = useState(false); const [showArchiveModal, setShowArchiveModal] = useState(false); - const [showLatex, setShowLatex] = useState(false); // Raw text by default for performance with large docs + const [showLatex, setShowLatex] = useState(true); const [isGeneratingPDF, setIsGeneratingPDF] = useState(false); const containerRef = useRef(null); const pdfDownloadAvailable = isPDFDownloadAvailable(capabilities); diff --git a/frontend/src/components/autonomous/LivePaperProgress.jsx b/frontend/src/components/autonomous/LivePaperProgress.jsx index 3fbc363..bde2245 100644 --- a/frontend/src/components/autonomous/LivePaperProgress.jsx +++ b/frontend/src/components/autonomous/LivePaperProgress.jsx @@ -198,7 +198,7 @@ const LivePaperProgress = ({ api, isCompiling, capabilities }) => { : prependDisclaimer(paperData.content, 'paper') } className="live-paper-latex-renderer" - defaultRaw={true} + defaultRaw={false} showToggle={true} /> </div> diff --git a/frontend/src/components/autonomous/LiveTier3Progress.jsx b/frontend/src/components/autonomous/LiveTier3Progress.jsx index fde5583..775948e 100644 --- a/frontend/src/components/autonomous/LiveTier3Progress.jsx +++ b/frontend/src/components/autonomous/LiveTier3Progress.jsx @@ -346,7 +346,7 @@ const LiveTier3Progress = ({ api, status, capabilities }) => { : prependDisclaimer(paperData.content, 'paper') } className="live-tier3-latex-renderer" - defaultRaw={true} + defaultRaw={false} showToggle={true} /> </div> diff --git a/frontend/src/components/autonomous/MathematicalProofs.css b/frontend/src/components/autonomous/MathematicalProofs.css index 6386800..2a9a94a 100644 --- a/frontend/src/components/autonomous/MathematicalProofs.css +++ b/frontend/src/components/autonomous/MathematicalProofs.css @@ -6,6 +6,68 @@ color: var(--text-primary, #e5e7eb); } +.math-proof-prompt-groups { + display: grid; + gap: 0.9rem; +} + +.proof-prompt-group { + overflow: hidden; + border: 1px solid var(--border-default, #374151); + border-radius: 0.85rem; + background: var(--surface-2, #111827); +} + +.proof-prompt-group-header { + width: 100%; + border: 0; + background: transparent; + color: var(--text-primary, #f3f4f6); + padding: 1rem 1.1rem; + display: flex; + align-items: center; + justify-content: space-between; + gap: 1rem; + text-align: left; + cursor: pointer; + font: inherit; +} + +.proof-prompt-group-header:hover { + background: rgba(99, 102, 241, 0.08); +} + +.proof-prompt-group-header:focus-visible { + outline: 2px solid #818cf8; + outline-offset: -2px; +} + +.proof-prompt-group-copy { + display: grid; + gap: 0.35rem; + min-width: 0; +} + +.proof-prompt-group-copy strong { + overflow-wrap: anywhere; +} + +.proof-prompt-group-copy span { + color: var(--text-secondary, #9ca3af); + font-size: 0.88rem; +} + +.proof-prompt-group-chevron { + flex: 0 0 auto; + color: var(--text-secondary, #9ca3af); +} + +.proof-prompt-group > .math-proofs-list, +.proof-prompt-group > .run-history-group-body { + border-top: 1px solid var(--border-default, #374151); + padding: 1rem; +} + .math-proofs-header { display: flex; justify-content: space-between; @@ -207,6 +269,11 @@ color: #e8a060; } +.math-proofs-filter--duplicate-novel.active { + background: rgba(168, 85, 247, 0.14); + color: #c4b5fd; +} + .math-proofs-filter:disabled { cursor: not-allowed; opacity: 0.55; @@ -248,6 +315,10 @@ border-color: rgba(96, 165, 250, 0.28); } +.math-proof-card.duplicate-novel { + border-color: rgba(168, 85, 247, 0.38); +} + .math-proof-card.platinum { position: relative; overflow: hidden; @@ -328,6 +399,12 @@ border-color: rgba(96, 165, 250, 0.35); } +.math-proof-badge.duplicate-novel { + color: #c4b5fd; + border-color: rgba(168, 85, 247, 0.45); + background: rgba(168, 85, 247, 0.08); +} + .math-proof-badge.platinum { position: relative; overflow: hidden; diff --git a/frontend/src/components/autonomous/MathematicalProofs.jsx b/frontend/src/components/autonomous/MathematicalProofs.jsx index 5605b50..e1da0dd 100644 --- a/frontend/src/components/autonomous/MathematicalProofs.jsx +++ b/frontend/src/components/autonomous/MathematicalProofs.jsx @@ -57,6 +57,9 @@ function createEmptyGraphState() { function getTierBadge(proof) { const tier = proof.novelty_tier; + if (tier === 'duplicate_novel') { + return { cardClass: 'duplicate-novel', badgeClass: 'duplicate-novel', label: 'Duplicate Novel Proof' }; + } if (tier === 'major_mathematical_discovery') { return { cardClass: 'platinum', badgeClass: 'platinum', label: 'Major Mathematical Discovery' }; } @@ -75,6 +78,10 @@ function getTierBadge(proof) { return { cardClass: 'known', badgeClass: 'known', label: 'Known Proof' }; } +function isPromptNovelProof(proof) { + return Boolean(proof?.novel && proof?.novelty_tier !== 'duplicate_novel'); +} + function isManualProofEvent(data = {}) { const sourceId = String(data.source_id || ''); const trigger = String(data.trigger || ''); @@ -157,10 +164,20 @@ function MathematicalProofs({ if (!selectedProofId) { return; } - setFilter('novel'); + const selectedProof = proofs.find((proof) => proof.proof_id === selectedProofId); + if (!selectedProof) { + return; + } + if (selectedProof.novelty_tier === 'duplicate_novel') { + setFilter('duplicate_novel'); + } else if (!isPromptNovelProof(selectedProof)) { + setFilter('all'); + } else { + setFilter('novel'); + } setViewMode('list'); setExpandedProofId(selectedProofId); - }, [selectedProofId]); + }, [proofs, selectedProofId]); const expandedDependencyState = expandedProofId ? dependencyStateByProofId[expandedProofId] : null; @@ -372,7 +389,8 @@ function MathematicalProofs({ }, [availableSources, manualSourceId, manualSourceType]); const counts = useMemo(() => { - const novel = proofs.filter((proof) => proof.novel).length; + const novel = proofs.filter(isPromptNovelProof).length; + const duplicateNovel = proofs.filter((proof) => proof.novelty_tier === 'duplicate_novel').length; const majorDiscovery = proofs.filter((proof) => proof.novelty_tier === 'major_mathematical_discovery').length; const discovery = proofs.filter((proof) => proof.novelty_tier === 'mathematical_discovery').length; const variant = proofs.filter((proof) => proof.novelty_tier === 'novel_variant').length; @@ -381,6 +399,7 @@ function MathematicalProofs({ return { total: proofs.length, novel, + duplicateNovel, known: proofs.length - novel, majorDiscovery, discovery, @@ -392,7 +411,10 @@ function MathematicalProofs({ const visibleProofs = useMemo(() => { if (filter === 'novel') { - return proofs.filter((proof) => proof.novel); + return proofs.filter(isPromptNovelProof); + } + if (filter === 'duplicate_novel') { + return proofs.filter((proof) => proof.novelty_tier === 'duplicate_novel'); } if (filter === 'major_mathematical_discovery') { return proofs.filter((proof) => proof.novelty_tier === 'major_mathematical_discovery'); @@ -540,6 +562,12 @@ function MathematicalProofs({ > Formalization ({counts.formulation || 0}) </button> + <button + className={`math-proofs-filter math-proofs-filter--duplicate-novel ${filter === 'duplicate_novel' ? 'active' : ''}`} + onClick={() => setFilter('duplicate_novel')} + > + Duplicate Novel ({counts.duplicateNovel || 0}) + </button> <button className={`math-proofs-filter ${filter === 'all' ? 'active' : ''}`} onClick={() => setFilter('all')} @@ -725,8 +753,11 @@ function MathematicalProofs({ Download .lean </button> <button + type="button" className="math-proof-expand" onClick={() => setExpandedProofId(isExpanded ? null : proof.proof_id)} + aria-expanded={isExpanded} + aria-controls={`proof-details-${proof.proof_id}`} > {isExpanded ? 'Hide Details' : 'View Details'} </button> @@ -740,7 +771,12 @@ function MathematicalProofs({ </div> {isExpanded && ( - <div className="math-proof-details"> + <div + id={`proof-details-${proof.proof_id}`} + className="math-proof-details" + role="region" + aria-label={`Details for ${proof.theorem_name || proof.theorem_statement}`} + > <div className="math-proof-actions"> <button type="button" diff --git a/frontend/src/components/autonomous/MathematicalProofs.test.jsx b/frontend/src/components/autonomous/MathematicalProofs.test.jsx new file mode 100644 index 0000000..763c81f --- /dev/null +++ b/frontend/src/components/autonomous/MathematicalProofs.test.jsx @@ -0,0 +1,44 @@ +import React from 'react'; +import { render, screen } from '@testing-library/react'; +import userEvent from '@testing-library/user-event'; +import MathematicalProofs from './MathematicalProofs'; + +const proof = { + proof_id: 'proof-1', + run_id: 'run-1', + user_prompt: 'Prove the prompt-level theorem', + theorem_statement: 'theorem prompt_level : True', + source_type: 'paper', + source_id: 'paper-1', + source_title: 'Paper one', + lean_code: 'theorem prompt_level : True := by trivial', + novelty_tier: 'mathematical_discovery', + novel: true, +}; + +function buildApi() { + return { + getProofs: vi.fn().mockResolvedValue({ proofs: [proof] }), + getProofStatus: vi.fn().mockResolvedValue({ lean4_enabled: false }), + getBrainstorms: vi.fn().mockResolvedValue({ brainstorms: [] }), + getPapers: vi.fn().mockResolvedValue({ papers: [] }), + }; +} + +test('shows active proofs directly without a prompt-level collapse', async () => { + const user = userEvent.setup(); + render(<MathematicalProofs api={buildApi()} />); + + expect(await screen.findByText(proof.theorem_statement)).toBeInTheDocument(); + expect(screen.queryByRole('button', { name: /Prove the prompt-level theorem/i })).not.toBeInTheDocument(); + expect(screen.queryByText(proof.user_prompt)).not.toBeInTheDocument(); + await user.click(screen.getByRole('button', { name: 'View Details' })); + expect(screen.getByText(proof.lean_code)).toBeInTheDocument(); +}); + +test('opens a selected active proof directly', async () => { + render(<MathematicalProofs api={buildApi()} selectedProofId="proof-1" />); + + expect(await screen.findByText(proof.theorem_statement)).toBeInTheDocument(); + expect(screen.getByText(proof.lean_code)).toBeInTheDocument(); +}); diff --git a/frontend/src/components/autonomous/ProofGraph.css b/frontend/src/components/autonomous/ProofGraph.css index d1175fa..81b87e2 100644 --- a/frontend/src/components/autonomous/ProofGraph.css +++ b/frontend/src/components/autonomous/ProofGraph.css @@ -119,6 +119,10 @@ stroke: rgba(96, 165, 250, 0.32); } +.proof-graph-node-group.duplicate-novel .proof-graph-node-frame { + stroke: rgba(168, 85, 247, 0.42); +} + .proof-graph-node-group.platinum .proof-graph-node-frame { stroke: rgba(226, 232, 240, 0.85); filter: drop-shadow(0 0 5px rgba(226, 232, 240, 0.45)); diff --git a/frontend/src/components/autonomous/ProofGraph.jsx b/frontend/src/components/autonomous/ProofGraph.jsx index 5fec97b..a33066c 100644 --- a/frontend/src/components/autonomous/ProofGraph.jsx +++ b/frontend/src/components/autonomous/ProofGraph.jsx @@ -3,6 +3,7 @@ import './ProofGraph.css'; function getGraphNodeTierClass(node) { const tier = node.novelty_tier; + if (tier === 'duplicate_novel') return 'duplicate-novel'; if (tier === 'major_mathematical_discovery') return 'platinum'; if (tier === 'mathematical_discovery') return 'gold'; if (tier === 'novel_variant') return 'silver'; diff --git a/frontend/src/components/autonomous/ProofLibrary.css b/frontend/src/components/autonomous/ProofLibrary.css index 0c96d4c..eb04d1d 100644 --- a/frontend/src/components/autonomous/ProofLibrary.css +++ b/frontend/src/components/autonomous/ProofLibrary.css @@ -1,3 +1,19 @@ +.proof-library .run-history-group-header.proof-prompt-group-header { + appearance: none; + width: 100%; + font: inherit; + color: inherit; +} + +.proof-library .proof-prompt-group-chevron { + margin-left: 0.5rem; + font-size: 0.85rem; +} + +.proof-library .proof-prompt-group > .run-history-group-body { + padding: 1rem; +} + /* ProofLibrary.css - Proof-specific overrides for the FinalAnswerLibrary layout */ .proof-library .library-header h2 { @@ -13,6 +29,10 @@ border-left: 3px solid #666; } +.proof-card--duplicate-novel { + border-left: 3px solid #8b5cf6; +} + /* Tier-specific card accents */ .proof-card--platinum { border-left: 3px solid #e2e8f0; @@ -42,6 +62,11 @@ color: #ccc; } +.proof-badge--duplicate-novel { + background: linear-gradient(135deg, #8b5cf6 0%, #c4b5fd 100%); + color: #140a2e; +} + /* Tier-specific badges */ .proof-badge--platinum { position: relative; diff --git a/frontend/src/components/autonomous/ProofLibrary.jsx b/frontend/src/components/autonomous/ProofLibrary.jsx index d620164..3651144 100644 --- a/frontend/src/components/autonomous/ProofLibrary.jsx +++ b/frontend/src/components/autonomous/ProofLibrary.jsx @@ -1,7 +1,9 @@ -import React, { useState, useEffect, useMemo } from 'react'; +import React, { useState, useEffect, useMemo, useRef } from 'react'; import { autonomousAPI, proofSearchAPI } from '../../services/api'; import { buildResearchRunGroups, formatRunPromptPreview } from '../../utils/researchRunHistory'; import { downloadTextFile } from '../../utils/downloadHelpers'; +import { classifyProofNovelty, getCanonicalProofIdentity, sanitizeDomId } from '../../utils/proofPresentation'; +import { readBooleanStorage } from '../../utils/safeStorage'; import './FinalAnswerLibrary.css'; import './ProofLibrary.css'; @@ -20,33 +22,72 @@ function truncate(text, maxLength = 220) { } function getTierBadge(proof) { - const tier = proof.novelty_tier; - if (tier === 'major_mathematical_discovery') { - return { cssClass: 'proof-badge--platinum', label: 'Major Mathematical Discovery' }; - } - if (tier === 'mathematical_discovery') { - return { cssClass: 'proof-badge--gold', label: 'Minor Mathematical Discovery' }; + const presentation = classifyProofNovelty(proof); + return { cssClass: presentation.badgeClass, label: presentation.label }; +} + +function getCardClass(proof) { + return classifyProofNovelty(proof).cardClass; +} + +function getProofCardId(proof) { + return getCanonicalProofIdentity(proof); +} + +function getProofRunKey(proof) { + return proof.run_id || proof.session_id || `orphan:${proof.proof_id}`; +} + +function matchesSelectedProof(proof, selectedProofId, selectedSessionId = '', selectedRunId = '') { + if (!selectedProofId) return false; + const proofIds = [ + proof.proof_id, + proof.library_id, + proof.search_id, + proof.lean_code_hash, + proof.theorem_statement_hash, + ].filter(Boolean).map(String); + if (!proofIds.includes(String(selectedProofId))) { + return false; } - if (tier === 'novel_variant') { - return { cssClass: 'proof-badge--silver', label: 'Novel Reformulation' }; + if (selectedRunId) { + return (proof.run_id || proof.session_id || '') === selectedRunId; } - if (tier === 'novel_formulation') { - return { cssClass: 'proof-badge--bronze', label: 'Novel Formalization' }; + return !selectedSessionId || !proof.session_id || proof.session_id === selectedSessionId; +} + +const FEDERATED_QUERY_STORAGE_KEY = 'proof_library_federated_query'; +const FEDERATED_CORPORA_STORAGE_KEY = 'proof_library_federated_corpora'; +const FEDERATED_VERIFIED_STORAGE_KEY = 'proof_library_federated_verified_only'; +const LOCAL_QUERY_STORAGE_KEY = 'proof_library_local_query'; +const LOCAL_CATEGORY_STORAGE_KEY = 'proof_library_local_category'; +const DEFAULT_CORPORA = ['moto', 'manual', 'leanoj', 'syntheticlib4']; + +function readStringStorage(key, fallback) { + try { + const value = localStorage.getItem(key); + return value === null ? fallback : value; + } catch { + return fallback; } - if (proof.novel) { - return { cssClass: 'proof-badge--gold', label: 'Novel' }; +} + +function readCorporaStorage() { + try { + const parsed = JSON.parse(localStorage.getItem(FEDERATED_CORPORA_STORAGE_KEY)); + const valid = Array.isArray(parsed) ? parsed.filter((id) => DEFAULT_CORPORA.includes(id)) : []; + return valid.length ? valid : DEFAULT_CORPORA; + } catch { + return DEFAULT_CORPORA; } - return { cssClass: 'proof-badge--known', label: 'Known' }; } -function getCardClass(proof) { - const tier = proof.novelty_tier; - if (tier === 'major_mathematical_discovery') return 'proof-card--platinum'; - if (tier === 'mathematical_discovery') return 'proof-card--gold'; - if (tier === 'novel_variant') return 'proof-card--silver'; - if (tier === 'novel_formulation') return 'proof-card--bronze'; - if (proof.novel) return 'proof-card--novel'; - return 'proof-card--known'; +function writeStorage(key, value) { + try { + localStorage.setItem(key, value); + } catch { + // Storage is optional; keep the current in-memory controls usable. + } } const PROOF_SEARCH_CORPORA = [ @@ -58,29 +99,45 @@ const PROOF_SEARCH_CORPORA = [ const PROOF_SEARCH_OPTIONS = [ { id: 'verified_only', label: 'Verified only' }, - { id: 'include_partial', label: 'Include partial artifacts' }, - { id: 'include_failed', label: 'Include failed attempts' }, +]; + +const PROOF_LIBRARY_FILTERS = [ + { id: 'novel', label: 'Novel Proofs' }, + { id: 'duplicate_novel', label: 'Duplicate Novel Proofs' }, + { id: 'not_novel', label: 'Not Novel Proofs' }, + { id: 'all', label: 'All Proofs' }, ]; export default function ProofLibrary({ proofScope = 'autonomous', title = 'Proof Library', description = 'All verified mathematical proofs generated across research sessions.', + selectedProofId = null, + selectedSessionId = '', + selectedRunId = '', }) { const [proofs, setProofs] = useState([]); + const [proofCounts, setProofCounts] = useState({}); const [sessionsResponse, setSessionsResponse] = useState(null); const [loading, setLoading] = useState(true); const [error, setError] = useState(null); const [expandedId, setExpandedId] = useState(null); const [expandedProof, setExpandedProof] = useState(null); + const activeDetailRequestRef = useRef(null); + const libraryRequestGenerationRef = useRef(0); + const searchRequestGenerationRef = useRef(0); + const searchDetailRequestGenerationRef = useRef(0); + const [expandedRunGroups, setExpandedRunGroups] = useState(() => new Set()); const [loadingContentId, setLoadingContentId] = useState(null); - const [searchTerm, setSearchTerm] = useState(''); - const [filterNovelty, setFilterNovelty] = useState('novel'); + const [searchTerm, setSearchTerm] = useState(() => readStringStorage(LOCAL_QUERY_STORAGE_KEY, '')); + const [filterNovelty, setFilterNovelty] = useState(() => { + const value = readStringStorage(LOCAL_CATEGORY_STORAGE_KEY, 'novel'); + return PROOF_LIBRARY_FILTERS.some(({ id }) => id === value) ? value : 'novel'; + }); const [proofSearchOverview, setProofSearchOverview] = useState(null); - const [proofSearchCorpora, setProofSearchCorpora] = useState(['moto', 'manual', 'leanoj', 'syntheticlib4']); - const [proofSearchVerifiedOnly, setProofSearchVerifiedOnly] = useState(true); - const [proofSearchIncludePartial, setProofSearchIncludePartial] = useState(false); - const [proofSearchIncludeFailed, setProofSearchIncludeFailed] = useState(false); + const [proofSearchCorpora, setProofSearchCorpora] = useState(readCorporaStorage); + const [proofSearchVerifiedOnly, setProofSearchVerifiedOnly] = useState(() => readBooleanStorage(FEDERATED_VERIFIED_STORAGE_KEY, true)); + const [proofSearchQuery, setProofSearchQuery] = useState(() => readStringStorage(FEDERATED_QUERY_STORAGE_KEY, '')); const [proofSearchResults, setProofSearchResults] = useState([]); const [proofSearchMessage, setProofSearchMessage] = useState(''); const [proofSearchLoading, setProofSearchLoading] = useState(false); @@ -88,13 +145,16 @@ export default function ProofLibrary({ const [proofSearchExpandedRecord, setProofSearchExpandedRecord] = useState(null); const loadProofLibrary = async () => { + const generation = ++libraryRequestGenerationRef.current; + activeDetailRequestRef.current = null; + setExpandedId(null); + setExpandedProof(null); try { setLoading(true); setError(null); - const novelOnly = filterNovelty === 'novel'; const [proofsResult, sessionsResult] = await Promise.allSettled([ - autonomousAPI.getProofLibrary(novelOnly, proofScope), + autonomousAPI.getProofLibrary(filterNovelty, proofScope), proofScope === 'manual' ? Promise.resolve(null) : autonomousAPI.getSessions(), ]); @@ -102,7 +162,9 @@ export default function ProofLibrary({ throw proofsResult.reason; } - setProofs(proofsResult.value.proofs || []); + if (generation !== libraryRequestGenerationRef.current) return; + setProofs((proofsResult.value.proofs || []).map((proof) => ({ ...proof, scope: proofScope }))); + setProofCounts(proofsResult.value.counts || {}); if (sessionsResult.status === 'fulfilled') { setSessionsResponse(sessionsResult.value); @@ -110,9 +172,10 @@ export default function ProofLibrary({ setSessionsResponse(null); } } catch (err) { + if (generation !== libraryRequestGenerationRef.current) return; setError(err.message || 'Failed to load proof library'); } finally { - setLoading(false); + if (generation === libraryRequestGenerationRef.current) setLoading(false); } }; @@ -120,6 +183,28 @@ export default function ProofLibrary({ loadProofLibrary(); }, [filterNovelty, proofScope]); + useEffect(() => { + writeStorage(LOCAL_QUERY_STORAGE_KEY, searchTerm); + }, [searchTerm]); + useEffect(() => { + writeStorage(LOCAL_CATEGORY_STORAGE_KEY, filterNovelty); + }, [filterNovelty]); + useEffect(() => { + writeStorage(FEDERATED_QUERY_STORAGE_KEY, proofSearchQuery); + }, [proofSearchQuery]); + useEffect(() => { + writeStorage(FEDERATED_CORPORA_STORAGE_KEY, JSON.stringify(proofSearchCorpora)); + }, [proofSearchCorpora]); + useEffect(() => { + writeStorage(FEDERATED_VERIFIED_STORAGE_KEY, JSON.stringify(proofSearchVerifiedOnly)); + }, [proofSearchVerifiedOnly]); + useEffect(() => () => { + libraryRequestGenerationRef.current += 1; + searchRequestGenerationRef.current += 1; + searchDetailRequestGenerationRef.current += 1; + activeDetailRequestRef.current = null; + }, []); + useEffect(() => { let cancelled = false; proofSearchAPI.getOverview() @@ -153,7 +238,7 @@ export default function ProofLibrary({ const runGroups = useMemo(() => { const pseudoPapers = filteredProofs.map((p) => ({ - session_id: p.session_id, + session_id: p.run_id || p.session_id || `orphan:${p.proof_id}`, paper_id: p.proof_id, created_at: p.created_at, user_prompt: p.user_prompt, @@ -169,7 +254,7 @@ export default function ProofLibrary({ const proofsBySession = useMemo(() => { const map = new Map(); for (const proof of filteredProofs) { - const sid = proof.session_id || 'unknown'; + const sid = getProofRunKey(proof); if (!map.has(sid)) map.set(sid, []); map.get(sid).push(proof); } @@ -177,26 +262,99 @@ export default function ProofLibrary({ }, [filteredProofs]); const handleExpand = async (proof) => { - const id = proof.library_id || proof.proof_id; + const id = getProofCardId(proof); if (expandedId === id) { + activeDetailRequestRef.current = null; setExpandedId(null); setExpandedProof(null); return; } setExpandedId(id); + activeDetailRequestRef.current = id; setLoadingContentId(id); try { - const fullProof = await autonomousAPI.getLibraryProof(proof.session_id, proof.proof_id, proofScope); - setExpandedProof(fullProof); + let fullProof; + try { + fullProof = await proofSearchAPI.getProof(proof.corpus || proofScope, proof.proof_id, { + searchId: proof.search_id || null, + runId: proof.run_id || null, + sessionId: proof.session_id || null, + }); + if (!fullProof) throw new Error('Canonical proof hydration unavailable'); + } catch { + fullProof = await autonomousAPI.getLibraryProof(proof.session_id, proof.proof_id, proofScope); + } + if (activeDetailRequestRef.current === id) setExpandedProof(fullProof); } catch { - setExpandedProof(proof); + if (activeDetailRequestRef.current === id) setExpandedProof(proof); } finally { - setLoadingContentId(null); + setLoadingContentId((current) => (current === id ? null : current)); } }; + useEffect(() => { + if (!selectedProofId || loading) return; + const visibleMatch = filteredProofs.find((proof) => matchesSelectedProof(proof, selectedProofId, selectedSessionId, selectedRunId)); + if (visibleMatch) return; + const hiddenNoveltyMatch = proofs.find((proof) => matchesSelectedProof(proof, selectedProofId, selectedSessionId, selectedRunId)); + if (hiddenNoveltyMatch && filterNovelty !== 'all') { + setFilterNovelty('all'); + } + }, [filterNovelty, filteredProofs, loading, proofs, selectedProofId, selectedRunId, selectedSessionId]); + + useEffect(() => { + if (!selectedProofId || loading) return; + const match = filteredProofs.find((proof) => matchesSelectedProof(proof, selectedProofId, selectedSessionId, selectedRunId)); + if (!match) return; + const id = getProofCardId(match); + const runKey = getProofRunKey(match); + setExpandedRunGroups((previous) => new Set(previous).add(runKey)); + setExpandedId(id); + setLoadingContentId(id); + let cancelled = false; + proofSearchAPI.getProof(match.corpus || proofScope, match.proof_id, { + searchId: match.search_id || null, + runId: match.run_id || null, + sessionId: match.session_id || null, + }) + .then((fullProof) => { + if (!fullProof) throw new Error('Canonical proof hydration unavailable'); + return fullProof; + }) + .catch(() => autonomousAPI.getLibraryProof(match.session_id, match.proof_id, proofScope)) + .then((fullProof) => { + if (!cancelled) setExpandedProof(fullProof); + }) + .catch(() => { + if (!cancelled) setExpandedProof(match); + }) + .finally(() => { + if (!cancelled) { + setLoadingContentId(null); + setTimeout(() => { + document.getElementById(sanitizeDomId(id, 'proof-card'))?.scrollIntoView({ behavior: 'smooth', block: 'center' }); + }, 0); + } + }); + return () => { + cancelled = true; + }; + }, [filteredProofs, loading, proofScope, selectedProofId, selectedRunId, selectedSessionId]); + + const toggleRunGroup = (runKey) => { + setExpandedRunGroups((previous) => { + const next = new Set(previous); + if (next.has(runKey)) next.delete(runKey); + else next.add(runKey); + return next; + }); + activeDetailRequestRef.current = null; + setExpandedId(null); + setExpandedProof(null); + }; + const handleDownloadLean = async (proof, event) => { event?.stopPropagation(); @@ -227,21 +385,20 @@ export default function ProofLibrary({ }; const handleUnifiedProofSearch = async (event) => { + const generation = ++searchRequestGenerationRef.current; + searchDetailRequestGenerationRef.current += 1; event?.preventDefault(); setProofSearchLoading(true); setProofSearchMessage(''); setProofSearchExpandedId(null); setProofSearchExpandedRecord(null); try { - const effectiveVerifiedOnly = proofSearchVerifiedOnly - && !proofSearchIncludePartial - && !proofSearchIncludeFailed; const response = await proofSearchAPI.search({ - query: '', + query: proofSearchQuery, corpora: proofSearchCorpora, - verified_only: effectiveVerifiedOnly, - include_partial: proofSearchIncludePartial, - include_failed: proofSearchIncludeFailed, + verified_only: proofSearchVerifiedOnly, + include_partial: false, + include_failed: false, dependency_names: [], novelty_filters: [], module_filters: [], @@ -249,17 +406,24 @@ export default function ProofLibrary({ limit: 7, hydrate_lean_code: false, }); + if (generation !== searchRequestGenerationRef.current) return; + searchDetailRequestGenerationRef.current += 1; setProofSearchResults(response.results || []); setProofSearchMessage(response.weak_result_warning || response.ranking_notes || ''); } catch (err) { + if (generation !== searchRequestGenerationRef.current) return; + searchDetailRequestGenerationRef.current += 1; setProofSearchResults([]); setProofSearchMessage(err.message || 'Proof search failed'); } finally { - setProofSearchLoading(false); + if (generation === searchRequestGenerationRef.current) setProofSearchLoading(false); } }; const handleProofSearchReindex = async () => { + searchDetailRequestGenerationRef.current += 1; + setProofSearchExpandedId(null); + setProofSearchExpandedRecord(null); setProofSearchLoading(true); setProofSearchMessage(''); try { @@ -274,28 +438,36 @@ export default function ProofLibrary({ }; const handleExpandProofSearchResult = async (record) => { - const id = record.search_id || `${record.corpus}:${record.proof_id}`; + const id = record.search_id || `${record.corpus}:${record.session_id || ''}:${record.proof_id}`; if (proofSearchExpandedId === id) { + searchDetailRequestGenerationRef.current += 1; setProofSearchExpandedId(null); setProofSearchExpandedRecord(null); return; } setProofSearchExpandedId(id); setProofSearchExpandedRecord(record); + const generation = ++searchDetailRequestGenerationRef.current; if (record.lean_code) { return; } try { const hydrated = await proofSearchAPI.getProof(record.corpus, record.proof_id, { + searchId: record.search_id || null, + runId: record.run_id || null, sessionId: record.session_id || null, }); - setProofSearchExpandedRecord(hydrated); + if (generation === searchDetailRequestGenerationRef.current) { + setProofSearchExpandedRecord(hydrated); + } } catch { - setProofSearchExpandedRecord(record); + if (generation === searchDetailRequestGenerationRef.current) setProofSearchExpandedRecord(record); } }; - const novelCount = proofs.filter((p) => p.novel).length; + const novelCount = proofCounts.novel ?? proofs.filter((p) => classifyProofNovelty(p).group === 'novel').length; + const duplicateNovelCount = proofCounts.duplicate_novel ?? proofs.filter((p) => classifyProofNovelty(p).group === 'duplicate_novel').length; + const notNovelCount = proofCounts.not_novel ?? proofs.filter((p) => classifyProofNovelty(p).group === 'not_novel').length; const totalCount = proofs.length; if (loading) { @@ -329,15 +501,10 @@ export default function ProofLibrary({ <h2>{title}</h2> <p>{description}</p> <div className="library-stats"> - {filterNovelty === 'novel' ? ( - <span className="stat-badge">{novelCount} Novel Proof{novelCount !== 1 ? 's' : ''}</span> - ) : ( - <> - <span className="stat-badge">{totalCount} Total Proof{totalCount !== 1 ? 's' : ''}</span> - <span className="stat-badge">{novelCount} Novel</span> - <span className="stat-badge">{totalCount - novelCount} Known</span> - </> - )} + <span className="stat-badge">{totalCount} Listed Proof{totalCount !== 1 ? 's' : ''}</span> + <span className="stat-badge">{novelCount} Novel</span> + <span className="stat-badge">{duplicateNovelCount} Duplicate Novel</span> + <span className="stat-badge">{notNovelCount} Not Novel</span> </div> </div> @@ -356,6 +523,15 @@ export default function ProofLibrary({ </div> <form className="proof-search-form" onSubmit={handleUnifiedProofSearch}> + <label htmlFor="federated-proof-search-query">Federated proof query</label> + <input + id="federated-proof-search-query" + className="search-input" + type="search" + value={proofSearchQuery} + onChange={(event) => setProofSearchQuery(event.target.value)} + placeholder="Search all enabled proof corpora..." + /> <fieldset className="proof-search-checklist"> <legend>Search Checklist</legend> <div className="proof-search-checklist__grid"> @@ -382,41 +558,9 @@ export default function ProofLibrary({ </label> ); } - if (option.id === 'include_partial') { - return ( - <label key={option.id} className="proof-search-checkbox"> - <input - type="checkbox" - checked={proofSearchIncludePartial} - onChange={(event) => { - setProofSearchIncludePartial(event.target.checked); - if (event.target.checked) setProofSearchVerifiedOnly(false); - }} - /> - <span>{option.label}</span> - </label> - ); - } - return ( - <label key={option.id} className="proof-search-checkbox"> - <input - type="checkbox" - checked={proofSearchIncludeFailed} - onChange={(event) => { - setProofSearchIncludeFailed(event.target.checked); - if (event.target.checked) setProofSearchVerifiedOnly(false); - }} - /> - <span>{option.label}</span> - </label> - ); + return null; })} </div> - {(proofSearchIncludePartial || proofSearchIncludeFailed) && proofSearchVerifiedOnly && ( - <p className="proof-search-checklist__note"> - Partial or failed artifact searches automatically disable verified-only filtering for the request. - </p> - )} </fieldset> <div className="proof-search-actions"> <button type="submit" className="refresh-button" disabled={proofSearchLoading}> @@ -443,6 +587,7 @@ export default function ProofLibrary({ <div className="proof-search-results"> {proofSearchResults.map((record) => { const id = record.search_id || `${record.corpus}:${record.proof_id}`; + const detailsId = sanitizeDomId(id, 'proof-search-details'); const isExpanded = proofSearchExpandedId === id; const expandedRecord = isExpanded ? (proofSearchExpandedRecord || record) : record; return ( @@ -451,6 +596,8 @@ export default function ProofLibrary({ type="button" className="proof-search-result-card__summary" onClick={() => handleExpandProofSearchResult(record)} + aria-expanded={isExpanded} + aria-controls={detailsId} > <div> <h4>{record.theorem_name || record.display_title || record.proof_id}</h4> @@ -464,8 +611,15 @@ export default function ProofLibrary({ <span>{record.source_kind}</span> {record.lean_code_hash && <span>Code hash: {record.lean_code_hash}</span>} </div> - {isExpanded && ( - <div className="proof-expanded-content proof-search-expanded"> + <div + id={detailsId} + className="proof-expanded-content proof-search-expanded" + role="region" + aria-label={`Details for ${record.theorem_name || record.proof_id}`} + hidden={!isExpanded} + > + {isExpanded && ( + <> <div className="proof-detail-section"> <h4>Description</h4> <p>{expandedRecord.proof_description || expandedRecord.formal_sketch || 'No description available.'}</p> @@ -498,8 +652,9 @@ export default function ProofLibrary({ This result is metadata-only. Hydration did not return full Lean code for this record. </div> )} + </> + )} </div> - )} </div> ); })} @@ -509,6 +664,7 @@ export default function ProofLibrary({ <div className="library-controls"> <input + aria-label="Filter this proof library" className="search-input" type="text" placeholder="Search by theorem name, statement, source, or research question..." @@ -516,18 +672,16 @@ export default function ProofLibrary({ onChange={(e) => setSearchTerm(e.target.value)} /> <div className="filter-buttons"> - <button - className={filterNovelty === 'novel' ? 'active' : ''} - onClick={() => setFilterNovelty('novel')} - > - Novel Only - </button> - <button - className={filterNovelty === 'all' ? 'active' : ''} - onClick={() => setFilterNovelty('all')} - > - All Proofs - </button> + {PROOF_LIBRARY_FILTERS.map((filter) => ( + <button + key={filter.id} + className={filterNovelty === filter.id ? 'active' : ''} + onClick={() => setFilterNovelty(filter.id)} + aria-pressed={filterNovelty === filter.id} + > + {filter.label} + </button> + ))} </div> </div> @@ -548,10 +702,19 @@ export default function ProofLibrary({ {runGroups.map((group) => { const sessionProofs = proofsBySession.get(group.sessionId) || []; if (sessionProofs.length === 0) return null; + const isRunExpanded = expandedRunGroups.has(group.sessionId); + const runRegionId = sanitizeDomId(group.sessionId, 'proof-run'); return ( - <div key={group.sessionId} className="run-history-group"> - <div className="run-history-group-header"> + <div key={group.sessionId} className={`run-history-group proof-prompt-group ${isRunExpanded ? 'expanded' : ''}`}> + <button + type="button" + className="run-history-group-header proof-prompt-group-header" + onClick={() => toggleRunGroup(group.sessionId)} + aria-expanded={isRunExpanded} + aria-controls={runRegionId} + aria-label={`${isRunExpanded ? 'Collapse' : 'Expand'} proof run ${formatRunPromptPreview(group.userPrompt)}`} + > <div className="run-history-group-heading"> <h3 className="run-history-group-title"> {formatRunPromptPreview(group.userPrompt)} @@ -570,23 +733,35 @@ export default function ProofLibrary({ {group.isLegacy && ( <span className="run-history-group-badge">Legacy</span> )} + <span className="proof-prompt-group-chevron" aria-hidden="true"> + {isRunExpanded ? '▲' : '▼'} + </span> </div> - </div> - - <div className="run-history-group-body"> + </button> + + <div + id={runRegionId} + className="run-history-group-body" + role="region" + aria-label={`Proofs for ${formatRunPromptPreview(group.userPrompt)}`} + hidden={!isRunExpanded} + > + {isRunExpanded && ( <div className="answer-list"> {sessionProofs.map((proof) => { - const id = proof.library_id || proof.proof_id; + const id = getProofCardId(proof); + const cardDomId = sanitizeDomId(id, 'proof-card'); + const detailsDomId = sanitizeDomId(id, 'proof-library-details'); const isExpanded = expandedId === id; return ( <div + id={cardDomId} key={id} className={`answer-card proof-card ${isExpanded ? 'expanded' : ''} ${getCardClass(proof)}`} > <div className="answer-header" - onClick={() => handleExpand(proof)} > <div className="answer-title-row"> <h4 className="answer-title proof-title"> @@ -600,7 +775,14 @@ export default function ProofLibrary({ > Download .lean </button> - <button className="expand-button"> + <button + type="button" + className="expand-button" + onClick={() => handleExpand(proof)} + aria-expanded={isExpanded} + aria-controls={detailsDomId} + aria-label={`${isExpanded ? 'Collapse' : 'Expand'} proof ${proof.theorem_name || proof.proof_id}`} + > {isExpanded ? '\u25B2' : '\u25BC'} </button> </div> @@ -640,9 +822,15 @@ export default function ProofLibrary({ </div> </div> - {isExpanded && ( - <div className="answer-content"> - {loadingContentId === id ? ( + <div + id={detailsDomId} + className="answer-content" + role="region" + aria-label={`Details for ${proof.theorem_name || proof.proof_id}`} + hidden={!isExpanded} + > + {isExpanded && ( + loadingContentId === id ? ( <div className="library-loading" style={{ padding: '20px' }}> <span className="library-loading__icon">↻</span> <span className="library-loading__text">Loading proof details...</span> @@ -717,13 +905,14 @@ export default function ProofLibrary({ )} </div> </div> - ) : null} + ) : null + )} </div> - )} </div> ); })} </div> + )} </div> </div> ); @@ -732,21 +921,90 @@ export default function ProofLibrary({ ) : ( <div className="answer-list"> {filteredProofs.map((proof) => { - const id = proof.library_id || proof.proof_id; + const id = getProofCardId(proof); + const cardDomId = sanitizeDomId(id, 'proof-card'); + const detailsDomId = sanitizeDomId(id, 'proof-library-details'); + const isExpanded = expandedId === id; return ( - <div key={id} className="answer-card proof-card"> - <div className="answer-header" onClick={() => handleExpand(proof)}> + <div + id={cardDomId} + key={id} + className={`answer-card proof-card ${isExpanded ? 'expanded' : ''} ${getCardClass(proof)}`} + > + <div className="answer-header"> <div className="answer-title-row"> - <h4 className="answer-title">{proof.theorem_name || proof.proof_id}</h4> - <button - type="button" - className="proof-header-download" - onClick={(event) => handleDownloadLean(proof, event)} - > - Download .lean - </button> + <h4 className="answer-title proof-title">{proof.theorem_name || proof.proof_id}</h4> + <div className="proof-card-actions"> + <button + type="button" + className="proof-header-download" + onClick={(event) => handleDownloadLean(proof, event)} + > + Download .lean + </button> + <button + type="button" + className="expand-button" + onClick={() => handleExpand(proof)} + aria-expanded={isExpanded} + aria-controls={detailsDomId} + aria-label={`${isExpanded ? 'Collapse' : 'Expand'} proof ${proof.theorem_name || proof.proof_id}`} + > + {isExpanded ? '\u25B2' : '\u25BC'} + </button> + </div> </div> + <div className="answer-metadata"> + <span className={`format-badge ${getTierBadge(proof).cssClass}`}> + {getTierBadge(proof).label} + </span> + <span className="word-count"> + {proof.solver || 'Lean 4'} + </span> + </div> + <p className="proof-statement"> + {truncate(proof.theorem_statement, 300)} + </p> </div> + <div + id={detailsDomId} + className="answer-content" + role="region" + aria-label={`Details for ${proof.theorem_name || proof.proof_id}`} + hidden={!isExpanded} + > + {isExpanded && ( + loadingContentId === id ? ( + <div className="library-loading" style={{ padding: '20px' }}> + <span className="library-loading__icon">↻</span> + <span className="library-loading__text">Loading proof details...</span> + </div> + ) : expandedProof ? ( + <div className="proof-expanded-content"> + <div className="proof-detail-section"> + <h4>Theorem Statement</h4> + <pre className="proof-code-block"> + {expandedProof.theorem_statement} + </pre> + </div> + {expandedProof.novelty_reasoning && ( + <div className="proof-detail-section"> + <h4>Novelty Assessment</h4> + <p>{expandedProof.novelty_reasoning}</p> + </div> + )} + {expandedProof.lean_code && ( + <div className="proof-detail-section"> + <h4>Lean 4 Source Code</h4> + <pre className="proof-code-block proof-lean-code"> + {expandedProof.lean_code} + </pre> + </div> + )} + </div> + ) : null + )} + </div> </div> ); })} diff --git a/frontend/src/components/autonomous/ProofLibrary.test.jsx b/frontend/src/components/autonomous/ProofLibrary.test.jsx index 2c05a0c..e3f1ca4 100644 --- a/frontend/src/components/autonomous/ProofLibrary.test.jsx +++ b/frontend/src/components/autonomous/ProofLibrary.test.jsx @@ -3,6 +3,7 @@ import { render, screen, waitFor } from '@testing-library/react'; import userEvent from '@testing-library/user-event'; import ProofLibrary from './ProofLibrary'; import { autonomousAPI, proofSearchAPI } from '../../services/api'; +import { buildResearchRunGroups } from '../../utils/researchRunHistory'; vi.mock('../../services/api', () => ({ autonomousAPI: { @@ -29,7 +30,9 @@ vi.mock('../../utils/downloadHelpers', () => ({ beforeEach(() => { vi.clearAllMocks(); - autonomousAPI.getProofLibrary.mockResolvedValue({ proofs: [] }); + buildResearchRunGroups.mockReturnValue([]); + Element.prototype.scrollIntoView = vi.fn(); + autonomousAPI.getProofLibrary.mockResolvedValue({ proofs: [], counts: {} }); autonomousAPI.getSessions.mockResolvedValue({ sessions: [] }); proofSearchAPI.getOverview.mockResolvedValue({ total_records: 99, @@ -79,18 +82,18 @@ test('submits unified proof-search checklist and hydrates SyntheticLib4 results' expect(await screen.findByText('Unified Proof Search')).toBeInTheDocument(); expect(screen.getByText('99 indexed')).toBeInTheDocument(); expect(screen.getByText('7 result cap')).toBeInTheDocument(); + expect(screen.queryByLabelText(/Include partial artifacts/i)).not.toBeInTheDocument(); + expect(screen.queryByLabelText(/Include failed attempts/i)).not.toBeInTheDocument(); - await user.click(screen.getByLabelText(/Include partial artifacts/i)); - await user.click(screen.getByLabelText(/Include failed attempts/i)); await user.click(screen.getByRole('button', { name: /Search Proofs/i })); await waitFor(() => { expect(proofSearchAPI.search).toHaveBeenCalledWith({ query: '', corpora: ['moto', 'manual', 'leanoj', 'syntheticlib4'], - verified_only: false, - include_partial: true, - include_failed: true, + verified_only: true, + include_partial: false, + include_failed: false, dependency_names: [], novelty_filters: [], module_filters: [], @@ -106,6 +109,8 @@ test('submits unified proof-search checklist and hydrates SyntheticLib4 results' await waitFor(() => { expect(proofSearchAPI.getProof).toHaveBeenCalledWith('syntheticlib4', 'sl4-proof-1', { + searchId: 'syntheticlib4:sl4-proof-1', + runId: null, sessionId: null, }); }); @@ -136,3 +141,112 @@ test('keeps proof-search errors visible and clears stale results', async () => { expect(await screen.findByText(/SyntheticLib4 search temporarily unavailable/i)).toBeInTheDocument(); expect(screen.queryByText('synthetic_comm_monoid')).not.toBeInTheDocument(); }); + +test('renders proof library category tabs and requests selected category', async () => { + const user = userEvent.setup(); + render(<ProofLibrary />); + + expect(await screen.findByRole('button', { name: /^Novel Proofs$/i })).toBeInTheDocument(); + expect(screen.getByRole('button', { name: /^Duplicate Novel Proofs$/i })).toBeInTheDocument(); + expect(screen.getByRole('button', { name: /^Not Novel Proofs$/i })).toBeInTheDocument(); + expect(screen.getByRole('button', { name: /^All Proofs$/i })).toBeInTheDocument(); + expect(autonomousAPI.getProofLibrary).toHaveBeenCalledWith('novel', 'autonomous'); + + await user.click(screen.getByRole('button', { name: /^Duplicate Novel Proofs$/i })); + + await waitFor(() => { + expect(autonomousAPI.getProofLibrary).toHaveBeenLastCalledWith('duplicate_novel', 'autonomous'); + }); +}); + +test('renders duplicate novel proof badge', async () => { + const user = userEvent.setup(); + autonomousAPI.getProofLibrary.mockResolvedValue({ + counts: { duplicate_novel: 1 }, + proofs: [ + { + proof_id: 'duplicate-novel-proof-1', + session_id: 'session-a', + theorem_name: 'DuplicateNovel.Helper', + theorem_statement: 'theorem duplicate_novel_helper : True', + source_title: 'Duplicate source', + novelty_tier: 'duplicate_novel', + novel: true, + created_at: '2026-06-12T00:00:00+00:00', + }, + ], + }); + buildResearchRunGroups.mockReturnValue([ + { + sessionId: 'session-a', + userPrompt: 'Duplicate proof run', + createdAt: '2026-06-12T00:00:00+00:00', + }, + ]); + + render(<ProofLibrary />); + + const groupButton = await screen.findByRole('button', { name: /Duplicate proof run/i }); + expect(groupButton).toHaveAttribute('aria-expanded', 'false'); + expect(screen.queryByText('DuplicateNovel.Helper')).not.toBeInTheDocument(); + await user.click(groupButton); + expect(await screen.findByText('DuplicateNovel.Helper')).toBeInTheDocument(); + expect(screen.getByText('Duplicate Novel')).toBeInTheDocument(); +}); + +test('selected proof id switches filters, expands, and hydrates matching proof history card', async () => { + autonomousAPI.getProofLibrary.mockResolvedValue({ + counts: { not_novel: 1 }, + proofs: [ + { + proof_id: 'known-proof-1', + session_id: 'session-a', + theorem_name: 'Known.Helper', + theorem_statement: 'theorem known_helper : True', + source_title: 'Known source', + novelty_tier: 'not_novel', + novel: false, + created_at: '2026-06-12T00:00:00+00:00', + }, + ], + }); + autonomousAPI.getSessions.mockResolvedValue({ + sessions: [ + { + session_id: 'session-a', + user_prompt: 'Known proof run', + created_at: '2026-06-12T00:00:00+00:00', + }, + ], + }); + autonomousAPI.getLibraryProof.mockResolvedValue({ + proof_id: 'known-proof-1', + session_id: 'session-a', + theorem_name: 'Known.Helper', + theorem_statement: 'theorem known_helper : True', + novelty_reasoning: 'Known but useful.', + lean_code: 'theorem known_helper : True := by trivial', + }); + proofSearchAPI.getProof.mockRejectedValueOnce(new Error('Legacy record is not indexed')); + buildResearchRunGroups.mockReturnValue([ + { + sessionId: 'session-a', + userPrompt: 'Known proof run', + createdAt: '2026-06-12T00:00:00+00:00', + }, + ]); + + render(<ProofLibrary selectedProofId="known-proof-1" selectedSessionId="session-a" />); + + expect(await screen.findByText('Known.Helper')).toBeInTheDocument(); + await waitFor(() => { + expect(proofSearchAPI.getProof).toHaveBeenCalledWith('autonomous', 'known-proof-1', { + searchId: null, + runId: null, + sessionId: 'session-a', + }); + expect(autonomousAPI.getLibraryProof).toHaveBeenCalledWith('session-a', 'known-proof-1', 'autonomous'); + }); + expect(await screen.findByText('Known but useful.')).toBeInTheDocument(); + expect(Element.prototype.scrollIntoView).toHaveBeenCalled(); +}); diff --git a/frontend/src/components/compiler/CompilerInterface.jsx b/frontend/src/components/compiler/CompilerInterface.jsx index 004166f..9a9adee 100644 --- a/frontend/src/components/compiler/CompilerInterface.jsx +++ b/frontend/src/components/compiler/CompilerInterface.jsx @@ -237,11 +237,13 @@ function CompilerInterface({ } }; - const handleTextFileLoaded = (content) => { - // Append to existing prompt with separator - const separator = compilerPrompt.trim() ? '\n\n' : ''; - const newPrompt = compilerPrompt + separator + content; - setCompilerPrompt(newPrompt); + const handleTextFileLoaded = (content, filename = 'uploaded-file') => { + const isLeanFile = filename.toLowerCase().endsWith('.lean'); + const labeledContext = `[UPLOADED ${isLeanFile ? 'LEAN FILE' : 'TEXT FILE'}: ${filename}]\n${content}`; + setCompilerPrompt(prevPrompt => { + const separator = prevPrompt.trim() ? '\n\n' : ''; + return prevPrompt + separator + labeledContext; + }); }; const handleStart = async () => { @@ -388,7 +390,7 @@ function CompilerInterface({ lean4_lsp_idle_timeout: status.lean4_lsp_idle_timeout ?? 600, max_parallel_candidates: status.proof_max_parallel_candidates ?? 6, smt_enabled: Boolean(status.smt_enabled), - smt_timeout: status.smt_timeout ?? 30, + smt_timeout: status.smt_timeout ?? 300, }); if (enabled) { const leanVersion = String(updatedStatus.lean4_version || updatedStatus.lean_version || '').trim(); @@ -450,65 +452,9 @@ function CompilerInterface({ <div> <h2>Single Paper Writer</h2> <p className="settings-hint"> - Compile the accepted aggregator database into one live mathematical paper. + Compile the accepted aggregator database into one live rigorous research paper or solution report. </p> </div> - <div className="autonomous-controls-stack"> - <div className="autonomous-controls"> - {!status.is_running ? ( - <button - onClick={handleStart} - className="btn-start" - disabled={isStarting || (anyWorkflowRunning && !status.is_running)} - > - {isStarting ? 'Starting...' : 'Start Writer'} - </button> - ) : ( - <> - <span className="runtime-indicator" role="status" aria-live="polite" title="Single paper writer is running"> - <span className="runtime-indicator-dot" aria-hidden="true"></span> - <span className="runtime-indicator-label">Running</span> - </span> - <button - onClick={handleStop} - className="btn-stop" - > - Stop Writer - </button> - </> - )} - </div> - <div - className="allowed-outputs-row" - title="Allowed Outputs controls which products this workflow may generate. At least one output must remain enabled." - > - <span className="allowed-outputs-label">Allowed Outputs:</span> - <label - className="allowed-output-option" - title="Mathematical Proofs enables Lean 4 proof verification and proof-library output for this run." - > - <input - type="checkbox" - checked={proofOutputsAvailable && Boolean(allowedOutputs.mathematicalProofs)} - onChange={(event) => updateAllowedOutput('mathematicalProofs', event.target.checked)} - disabled={status.is_running || proofOutputUpdating || !proofOutputsAvailable} - /> - <span className="allowed-output-text">Mathematical Proofs</span> - </label> - <label - className="allowed-output-option" - title="Research Papers enables the Single Paper Writer compilation output. When disabled, the writer runs proof extraction over the aggregator database instead of compiling a paper." - > - <input - type="checkbox" - checked={Boolean(allowedOutputs.researchPapers)} - onChange={(event) => updateAllowedOutput('researchPapers', event.target.checked)} - disabled={status.is_running} - /> - <span className="allowed-output-text">Research Papers</span> - </label> - </div> - </div> </div> <div className="status-section"> @@ -579,25 +525,85 @@ function CompilerInterface({ </div> )} - <div className="research-prompt-section"> - <label htmlFor="compilerPrompt">Compiler-Directing Prompt:</label> - <textarea - id="compilerPrompt" - value={compilerPrompt} - onChange={(e) => setCompilerPrompt(e.target.value)} - placeholder='Create a final prompt that exactly relates to a solution your aggregation database helps solve, i.e. "Tell me the most theoretically advanced perspective on the "squaring the circle" problem."' - rows={6} - disabled={status.is_running} - /> - <TextFileUploader - onFileLoaded={handleTextFileLoaded} - disabled={status.is_running} - maxSizeMB={5} - showCharCount={true} - confirmIfNotEmpty={true} - existingPromptLength={compilerPrompt.length} - /> - <small>This prompt directs the compiler on what kind of mathematical document to create from the aggregated database. View your in-progress and final answer in the "Live Paper" tab.</small> + <div className="research-prompt-section prompt-composer-section"> + <div className="prompt-composer"> + <label htmlFor="compilerPrompt">Compiler-Directing Prompt:</label> + <textarea + id="compilerPrompt" + value={compilerPrompt} + onChange={(e) => setCompilerPrompt(e.target.value)} + placeholder='Create a final prompt that exactly relates to a solution your aggregation database helps solve, i.e. "Tell me the most theoretically advanced perspective on the "squaring the circle" problem."' + rows={6} + disabled={status.is_running} + /> + <div className="prompt-composer-actions"> + <TextFileUploader + onFileLoaded={handleTextFileLoaded} + disabled={status.is_running} + maxSizeMB={5} + showCharCount={true} + confirmIfNotEmpty={true} + existingPromptLength={compilerPrompt.length} + /> + {!status.is_running ? ( + <button + type="button" + onClick={handleStart} + className="btn-start prompt-composer-primary" + disabled={isStarting || (anyWorkflowRunning && !status.is_running)} + > + {isStarting ? 'Starting...' : 'Start Writer'} + </button> + ) : ( + <div className="prompt-composer-running-actions"> + <span className="runtime-indicator" role="status" aria-live="polite" title="Single paper writer is running"> + <span className="runtime-indicator-dot" aria-hidden="true"></span> + <span className="runtime-indicator-label">Running</span> + </span> + <button + type="button" + onClick={handleStop} + className="btn-stop" + > + Stop Writer + </button> + </div> + )} + </div> + </div> + <div className="prompt-composer-options"> + <div + className="allowed-outputs-row" + title="Allowed Outputs controls which products this workflow may generate. At least one output must remain enabled." + > + <span className="allowed-outputs-label">Allowed Outputs:</span> + <label + className="allowed-output-option" + title="Mathematical Proofs enables Lean 4 proof verification and proof-library output for this run." + > + <input + type="checkbox" + checked={proofOutputsAvailable && Boolean(allowedOutputs.mathematicalProofs)} + onChange={(event) => updateAllowedOutput('mathematicalProofs', event.target.checked)} + disabled={status.is_running || proofOutputUpdating || !proofOutputsAvailable} + /> + <span className="allowed-output-text">Mathematical Proofs</span> + </label> + <label + className="allowed-output-option" + title="Research Papers enables the Single Paper Writer compilation output. When disabled, the writer runs proof extraction over the aggregator database instead of compiling a paper." + > + <input + type="checkbox" + checked={Boolean(allowedOutputs.researchPapers)} + onChange={(event) => updateAllowedOutput('researchPapers', event.target.checked)} + disabled={status.is_running} + /> + <span className="allowed-output-text">Research Papers</span> + </label> + </div> + </div> + <small>This prompt directs the compiler on what kind of rigorous paper or solution report to create from the aggregated database. View your in-progress and final answer in the "Live Paper" tab.</small> </div> <div className="stats-section"> diff --git a/frontend/src/components/compiler/CompilerInterface.test.jsx b/frontend/src/components/compiler/CompilerInterface.test.jsx new file mode 100644 index 0000000..0c24619 --- /dev/null +++ b/frontend/src/components/compiler/CompilerInterface.test.jsx @@ -0,0 +1,74 @@ +import { render, screen, waitFor } from '@testing-library/react'; +import { beforeEach, expect, test, vi } from 'vitest'; +import CompilerInterface from './CompilerInterface'; +import { compilerAPI } from '../../services/api'; + +vi.mock('../../services/api', () => ({ + autonomousAPI: { + getProofStatus: vi.fn(), + updateProofSettings: vi.fn(), + }, + compilerAPI: { + getStatus: vi.fn(), + getPrompt: vi.fn(), + start: vi.fn(), + stop: vi.fn(), + }, +})); + +vi.mock('../../services/websocket', () => ({ + websocket: { + on: vi.fn(), + off: vi.fn(), + }, +})); + +vi.mock('../TextFileUploader', () => ({ + default: () => null, +})); + +beforeEach(() => { + vi.clearAllMocks(); + localStorage.clear(); + sessionStorage.clear(); + compilerAPI.getStatus.mockResolvedValue({ + data: { + is_running: false, + current_mode: 'idle', + }, + }); + compilerAPI.getPrompt.mockResolvedValue({ + data: { + prompt: 'Write a focused paper from the aggregator database.', + }, + }); +}); + +test('keeps single paper writer start and output controls available in the prompt composer', async () => { + render( + <CompilerInterface + activeTab="compiler-interface" + capabilities={{ genericMode: false, lmStudioEnabled: true }} + anyWorkflowRunning={false} + onWorkflowRunningChange={vi.fn()} + connectivityStatus={{ + skills: { + agent_conversation_memory: { enabled: true }, + }, + }} + /> + ); + + expect(await screen.findByLabelText('Compiler-Directing Prompt:')).toBeInTheDocument(); + expect( + screen.getByText(/one live rigorous research paper or solution report/i) + ).toBeInTheDocument(); + expect( + screen.getByText(/what kind of rigorous paper or solution report/i) + ).toBeInTheDocument(); + await waitFor(() => { + expect(screen.getByRole('button', { name: 'Start Writer' })).toBeEnabled(); + }); + expect(screen.getByLabelText('Mathematical Proofs')).toBeInTheDocument(); + expect(screen.getByLabelText('Research Papers')).toBeInTheDocument(); +}); diff --git a/frontend/src/components/compiler/CompilerLogs.jsx b/frontend/src/components/compiler/CompilerLogs.jsx index 7f50967..68bb91f 100644 --- a/frontend/src/components/compiler/CompilerLogs.jsx +++ b/frontend/src/components/compiler/CompilerLogs.jsx @@ -5,16 +5,21 @@ import LiveActivityFeed from '../LiveActivityFeed'; import { formatContextOverflowActivityMessage, formatAssistantProofPackEventMessage, + formatSolutionPathEventMessage, getActivityClass, getActivityIcon, hasRecentAssistantProofPackDuplicate, } from '../../utils/activityStyles'; import { - MANUAL_AGGREGATOR_PROOF_SOURCE_ID, MANUAL_COMPILER_CURRENT_PROOF_SOURCE_ID, } from '../../hooks/useProofCheckRuntime'; +import { getNamespacedStorageKey } from '../../utils/runtimeConfig'; import '../autonomous/AutonomousResearch.css'; +const MAX_COMPILER_ACTIVITY_EVENTS = 2000; +const MAX_PERSISTED_TEXT_LENGTH = 1200; +export const COMPILER_ACTIVITY_STORAGE_KEY = getNamespacedStorageKey('compiler_events_log'); + const MANUAL_PROOF_EVENTS = [ 'proof_check_started', 'proof_check_no_candidates', @@ -32,6 +37,57 @@ const MANUAL_PROOF_EVENTS = [ 'proof_check_complete', ]; +export const shouldIncludeCompilerContextOverflow = (data = {}) => { + const roleId = String(data.role_id || '').toLowerCase(); + const workflowMode = String(data.workflow_mode || '').toLowerCase(); + return !( + (workflowMode && workflowMode !== 'compiler') + || (!workflowMode && !roleId.startsWith('compiler_')) + ); +}; + +export const shouldIncludeCompilerProofContextOverflow = (data = {}) => ( + data.source_type === 'paper' + && ( + data.source_id === MANUAL_COMPILER_CURRENT_PROOF_SOURCE_ID + || String(data.source_id || '').startsWith('manual_compiler_') + || String(data.source_id || '').startsWith('compiler_manual_') + ) +); + +export const shouldIncludeCompilerSolutionPathEvent = (data = {}) => { + const workflowMode = String(data.workflow_mode || data.mode || '').toLowerCase(); + return !workflowMode || workflowMode === 'compiler'; +}; + +const compactPersistedValue = (value) => { + if (typeof value === 'string') { + return value.length > MAX_PERSISTED_TEXT_LENGTH + ? `${value.slice(0, MAX_PERSISTED_TEXT_LENGTH)}...` + : value; + } + if (Array.isArray(value)) { + return value.slice(0, 20).map(compactPersistedValue); + } + if (value && typeof value === 'object') { + return Object.fromEntries( + Object.entries(value) + .slice(0, 40) + .map(([key, nested]) => [key, compactPersistedValue(nested)]) + ); + } + return value; +}; + +export const compactCompilerActivityEvents = (events = []) => ( + events.slice(0, MAX_COMPILER_ACTIVITY_EVENTS).map((event) => ({ + type: event.type, + timestamp: event.timestamp, + fullTimestamp: event.fullTimestamp, + data: compactPersistedValue(event.data || {}), + })) +); + const compactProofText = (value, maxLength = 1800) => { const cleaned = String(value || '').replace(/\s+/g, ' ').trim(); if (!cleaned) { @@ -129,14 +185,20 @@ function CompilerLogs() { }; const handleContextOverflow = (data) => { - const roleId = String(data?.role_id || '').toLowerCase(); - if (roleId && !roleId.startsWith('compiler_')) { + if (!shouldIncludeCompilerContextOverflow(data)) { return; } addEvent({ type: 'context_overflow_error', data }); loadStatus(); }; + const handleProofContextOverflow = (data = {}) => { + if (!shouldIncludeCompilerProofContextOverflow(data)) { + return; + } + addEvent({ type: 'proof_context_overflow', data }); + }; + const handleCorruptionDetected = (data) => { addEvent({ type: 'model_corruption_detected', data: { message: `Model ${data.model_id} corrupted (${data.failure_count} failures)` } }); }; @@ -161,9 +223,7 @@ function CompilerLogs() { const handleManualProofEvent = (eventName, data = {}) => { const isCurrentPaperProof = data.source_type === 'paper' && data.source_id === MANUAL_COMPILER_CURRENT_PROOF_SOURCE_ID; - const isAggregatorProofOnly = data.source_type === 'brainstorm' - && data.source_id === MANUAL_AGGREGATOR_PROOF_SOURCE_ID; - if (!isCurrentPaperProof && !isAggregatorProofOnly) { + if (!isCurrentPaperProof) { return; } addEvent({ type: eventName, data }); @@ -220,6 +280,7 @@ function CompilerLogs() { websocket.on('compiler_error', handleCompilerError); websocket.on('compiler_warning', handleCompilerWarning); websocket.on('context_overflow_error', handleContextOverflow); + websocket.on('proof_context_overflow', handleProofContextOverflow); websocket.on('model_corruption_detected', handleCorruptionDetected); websocket.on('model_recovery_initiated', handleRecoveryInitiated); websocket.on('model_recovery_success', handleRecoverySuccess); @@ -227,6 +288,22 @@ function CompilerLogs() { websocket.on('hung_connection_alert', handleHungConnectionAlert); websocket.on('assistant_proof_pack_updated', handleAssistantProofPackUpdated); websocket.on('assistant_proof_pack_failed', handleAssistantProofPackFailed); + const solutionPathEvents = [ + 'solution_path_activated', + 'solution_path_proposal_queued', + 'solution_path_proposal_reviewing', + 'solution_path_updated', + 'solution_path_proposal_rejected', + 'solution_path_proposal_retry_queued', + 'solution_path_proposal_user_repair_required', + 'solution_path_proposal_resumed', + ]; + const solutionPathUnsubscribers = solutionPathEvents.map((eventName) => ( + websocket.on(eventName, (data = {}) => { + if (!shouldIncludeCompilerSolutionPathEvent(data)) return; + addEvent({ type: eventName, data }); + }) + )); // Critique phase events websocket.on('critique_phase_started', handleCritiquePhaseStarted); @@ -262,6 +339,7 @@ function CompilerLogs() { websocket.off('compiler_error', handleCompilerError); websocket.off('compiler_warning', handleCompilerWarning); websocket.off('context_overflow_error', handleContextOverflow); + websocket.off('proof_context_overflow', handleProofContextOverflow); websocket.off('model_corruption_detected', handleCorruptionDetected); websocket.off('model_recovery_initiated', handleRecoveryInitiated); websocket.off('model_recovery_success', handleRecoverySuccess); @@ -289,6 +367,7 @@ function CompilerLogs() { websocket.off('compiler_wolfram_call', handleCompilerEvent); manualProofUnsubscribers.forEach((unsubscribe) => unsubscribe()); + solutionPathUnsubscribers.forEach((unsubscribe) => unsubscribe()); }; }, []); @@ -335,11 +414,14 @@ function CompilerLogs() { if (hasRecentAssistantProofPackDuplicate(prev, newEvent.type, newEvent.data || {}, fullTimestamp)) { return prev; } - const updated = [newEvent, ...prev].slice(0, 10000); // Keep last 10k events + const updated = [newEvent, ...prev].slice(0, MAX_COMPILER_ACTIVITY_EVENTS); // Save to localStorage for persistence try { - localStorage.setItem('compiler_events_log', JSON.stringify(updated)); + localStorage.setItem( + COMPILER_ACTIVITY_STORAGE_KEY, + JSON.stringify(compactCompilerActivityEvents(updated)) + ); } catch (e) { console.error('Failed to save events to localStorage:', e); } @@ -359,9 +441,10 @@ function CompilerLogs() { // Load events from localStorage on mount useEffect(() => { try { - const savedEvents = localStorage.getItem('compiler_events_log'); + const savedEvents = localStorage.getItem(COMPILER_ACTIVITY_STORAGE_KEY); if (savedEvents) { - setEvents(JSON.parse(savedEvents)); + const parsed = JSON.parse(savedEvents); + setEvents(Array.isArray(parsed) ? parsed.slice(0, MAX_COMPILER_ACTIVITY_EVENTS) : []); } } catch (e) { console.error('Failed to load events from localStorage:', e); @@ -374,7 +457,7 @@ function CompilerLogs() { const clearEventsLog = () => { setEvents([]); - localStorage.removeItem('compiler_events_log'); + localStorage.removeItem(COMPILER_ACTIVITY_STORAGE_KEY); }; // Format event data for user-friendly display @@ -413,6 +496,9 @@ function CompilerLogs() { if (type === 'assistant_proof_pack_updated' || type === 'assistant_proof_pack_failed') { return formatAssistantProofPackEventMessage(type, data); } + if (type.startsWith('solution_path_')) { + return formatSolutionPathEventMessage(type, data); + } // Phase transitions if (type === 'phase_transition') { @@ -435,7 +521,7 @@ function CompilerLogs() { if (type === 'compiler_decline') { return `Declined: ${data.mode || 'unknown'} - ${(data.reasoning || '').substring(0, 60)}...`; } - if (type === 'context_overflow_error') { + if (type === 'context_overflow_error' || type === 'proof_context_overflow') { return formatContextOverflowActivityMessage(data); } if (type === 'paper_updated') { @@ -458,10 +544,22 @@ function CompilerLogs() { return 'Proof check started for the current manual Compiler paper'; } if (type === 'proof_check_no_candidates') { - return 'No formal theorem candidates found in the current manual Compiler paper'; + const round = Number(data.proof_round_index || 0); + const maxRounds = Number(data.proof_max_rounds || 0); + const prefix = round > 0 && maxRounds > 1 + ? `Proof round ${round}/${maxRounds} discovery` + : 'Proof discovery'; + return `${prefix} found 0 proof candidates; no proofs will be attempted`; } if (type === 'proof_check_candidates_found') { - return `Proof candidates found: ${data.count || 0}`; + const count = Number(data.count || 0); + const round = Number(data.proof_round_index || 0); + const maxRounds = Number(data.proof_max_rounds || 0); + const prefix = round > 0 && maxRounds > 1 + ? `Proof round ${round}/${maxRounds} discovery` + : 'Proof discovery'; + const subject = count === 1 ? 'proof candidate' : 'proof candidates'; + return `${prefix} found ${count} ${subject}; ${count} will be attempted`; } if (type === 'proof_attempt_started') { return `Lean proof attempt started: ${proofTargetLabel(data)}`; diff --git a/frontend/src/components/compiler/CompilerSettings.jsx b/frontend/src/components/compiler/CompilerSettings.jsx index 0c98e9c..e05ac0a 100644 --- a/frontend/src/components/compiler/CompilerSettings.jsx +++ b/frontend/src/components/compiler/CompilerSettings.jsx @@ -30,6 +30,7 @@ import { } from '../../utils/oauthProviders'; import HelpTooltip from '../HelpTooltip'; import HighlightedModelsSidebar from '../HighlightedModelsSidebar'; +import OpenRouterFreeModelsControl from '../OpenRouterFreeModelsControl'; import ProofStrengthBadge from '../ProofStrengthBadge'; import RawSettingsEditor from '../RawSettingsEditor'; import '../autonomous/AutonomousResearch.css'; @@ -78,8 +79,8 @@ function CompilerSettings({ const [sakanaFuguModelError, setSakanaFuguModelError] = useState(''); const [loadingModels, setLoadingModels] = useState(true); const [freeOnly, setFreeOnly] = useState(false); - const [freeModelLooping, setFreeModelLooping] = useState(true); - const [freeModelAutoSelector, setFreeModelAutoSelector] = useState(true); + const [freeModelLooping, setFreeModelLooping] = useState(false); + const [freeModelAutoSelector, setFreeModelAutoSelector] = useState(false); // Validator settings const [validatorProvider, setValidatorProvider] = useState('lm_studio'); @@ -323,8 +324,8 @@ function CompilerSettings({ try { const freeModelSettings = await openRouterAPI.getFreeModelSettings(); - setFreeModelLooping(freeModelSettings.looping_enabled ?? true); - setFreeModelAutoSelector(freeModelSettings.auto_selector_enabled ?? true); + setFreeModelLooping(freeModelSettings.looping_enabled ?? false); + setFreeModelAutoSelector(freeModelSettings.auto_selector_enabled ?? false); } catch (error) { console.error('Failed to load free model settings:', error); } @@ -928,11 +929,11 @@ Be honest and constructive. Identify both strengths and weaknesses.`; setCritiqueSubmitterMaxOutput(rawSettings.critiqueSubmitterMaxOutput ?? DEFAULT_MAX_OUTPUT_TOKENS); setCritiqueSubmitterSuperchargeEnabled(Boolean(rawSettings.critiqueSubmitterSuperchargeEnabled)); setFreeOnly(rawSettings.freeOnly ?? false); - setFreeModelLooping(rawSettings.freeModelLooping ?? true); - setFreeModelAutoSelector(rawSettings.freeModelAutoSelector ?? true); + setFreeModelLooping(rawSettings.freeModelLooping ?? false); + setFreeModelAutoSelector(rawSettings.freeModelAutoSelector ?? false); setModelProviders(rawSettings.modelProviders || {}); openRouterAPI - .setFreeModelSettings(rawSettings.freeModelLooping ?? true, rawSettings.freeModelAutoSelector ?? true) + .setFreeModelSettings(rawSettings.freeModelLooping ?? false, rawSettings.freeModelAutoSelector ?? false) .catch(() => {}); if (updateRawText) { @@ -959,8 +960,8 @@ Be honest and constructive. Identify both strengths and weaknesses.`; critiqueSubmitterModel: rawSettings.critiqueSubmitterModel || '', critiqueSubmitterOpenrouterReasoningEffort: normalizeOpenRouterReasoningEffort(rawSettings.critiqueSubmitterOpenrouterReasoningEffort), freeOnly: rawSettings.freeOnly ?? false, - freeModelLooping: rawSettings.freeModelLooping ?? true, - freeModelAutoSelector: rawSettings.freeModelAutoSelector ?? true, + freeModelLooping: rawSettings.freeModelLooping ?? false, + freeModelAutoSelector: rawSettings.freeModelAutoSelector ?? false, modelProviders: rawSettings.modelProviders || {}, })); } @@ -1376,17 +1377,15 @@ Be honest and constructive. Identify both strengths and weaknesses.`; > 🔗 OpenRouter Model List </button> - <label className="settings-checkbox-label model-refresh-controls__toggle"> - <input - type="checkbox" - checked={freeOnly} - onChange={(e) => setFreeOnly(e.target.checked)} - /> - Free models only - </label> + <OpenRouterFreeModelsControl + checked={freeOnly} + onChange={setFreeOnly} + /> </> )} - {developerModeEnabled ? ( + {developerModeEnabled && ( + <> + {hasOpenRouterKey && <span className="model-refresh-controls__divider" aria-hidden="true" />} <label className="settings-checkbox-label model-refresh-controls__toggle"> <input type="checkbox" @@ -1395,10 +1394,7 @@ Be honest and constructive. Identify both strengths and weaknesses.`; /> Edit Raw </label> - ) : ( - <span className="settings-developer-mode-hint"> - Developer mode: press Shift + Z + X to toggle raw JSON settings. - </span> + </> )} </div> @@ -1440,7 +1436,7 @@ Be honest and constructive. Identify both strengths and weaknesses.`; {renderRoleConfig({ title: 'Assistant', - description: 'Runs in parallel during outline, writing, review, and proof work to retrieve up to 7 relevant memory supports from Session History Memory and SyntheticLib4 when enabled. Validators and critique phases do not receive Assistant context.', + description: 'Runs in parallel during outline, writing, review, and proof work to retrieve up to 7 relevant verified proof-memory supports from Session History Memory and SyntheticLib4 when enabled. Validators and critique phases do not receive Assistant context.', provider: assistantProvider, setProvider: setAssistantProvider, model: assistantModel || validatorModel, diff --git a/frontend/src/components/compiler/CompilerSettings.test.jsx b/frontend/src/components/compiler/CompilerSettings.test.jsx index d50cbc6..403c393 100644 --- a/frontend/src/components/compiler/CompilerSettings.test.jsx +++ b/frontend/src/components/compiler/CompilerSettings.test.jsx @@ -53,8 +53,8 @@ beforeEach(() => { localStorage.clear(); openRouterAPI.getApiKeyStatus.mockResolvedValue({ has_key: true }); openRouterAPI.getFreeModelSettings.mockResolvedValue({ - looping_enabled: true, - auto_selector_enabled: true, + looping_enabled: false, + auto_selector_enabled: false, }); openRouterAPI.getModels.mockResolvedValue({ models: [ diff --git a/frontend/src/components/contextOverflowBehavior.test.jsx b/frontend/src/components/contextOverflowBehavior.test.jsx new file mode 100644 index 0000000..b3ae56a --- /dev/null +++ b/frontend/src/components/contextOverflowBehavior.test.jsx @@ -0,0 +1,320 @@ +import React from 'react'; +import { act, cleanup, render, screen, waitFor } from '@testing-library/react'; +import { afterEach, beforeEach, describe, expect, test, vi } from 'vitest'; +import { + compactLiveActivityEvent, + isProviderNotificationDismissed, + persistDismissedProviderNotificationId, + readPersistedLiveActivity, + shouldRecordWorkflowStoppedActivity, +} from '../App'; +import { + formatAggregatorPersistedOverflowMessage, + shouldIncludeAggregatorProofContextOverflow, + shouldIncludeAggregatorSolutionPathEvent, +} from './aggregator/AggregatorLogs'; +import { + compactCompilerActivityEvents, + shouldIncludeCompilerContextOverflow, + shouldIncludeCompilerProofContextOverflow, + shouldIncludeCompilerSolutionPathEvent, +} from './compiler/CompilerLogs'; +import { formatContextOverflowActivityMessage } from '../utils/activityStyles'; +import AggregatorLogs from './aggregator/AggregatorLogs'; +import CompilerLogs from './compiler/CompilerLogs'; +import { api, compilerAPI } from '../services/api'; +import { websocket } from '../services/websocket'; + +describe('context overflow activity behavior', () => { + beforeEach(() => { + localStorage.clear(); + }); + + afterEach(() => { + cleanup(); + vi.restoreAllMocks(); + }); + + test('solution path activity stays in its owning manual workflow log', () => { + expect(shouldIncludeAggregatorSolutionPathEvent({ workflow_mode: 'aggregator' })).toBe(true); + expect(shouldIncludeAggregatorSolutionPathEvent({ workflow_mode: 'compiler' })).toBe(false); + expect(shouldIncludeAggregatorSolutionPathEvent({ workflow_mode: 'autonomous' })).toBe(false); + expect(shouldIncludeCompilerSolutionPathEvent({ workflow_mode: 'compiler' })).toBe(true); + expect(shouldIncludeCompilerSolutionPathEvent({ workflow_mode: 'aggregator' })).toBe(false); + expect(shouldIncludeCompilerSolutionPathEvent({ workflow_mode: 'leanoj' })).toBe(false); + }); + + test('Compiler accepts its own events and rejects autonomous, LeanOJ, and Aggregator events', () => { + expect(shouldIncludeCompilerContextOverflow({ + workflow_mode: 'compiler', + role_id: 'compiler_writer', + })).toBe(true); + expect(shouldIncludeCompilerContextOverflow({ + role_id: 'compiler_validator', + })).toBe(true); + expect(shouldIncludeCompilerContextOverflow({ + workflow_mode: 'autonomous', + role_id: 'compiler_writer', + })).toBe(false); + expect(shouldIncludeCompilerContextOverflow({ + workflow_mode: 'leanoj', + role_id: 'leanoj_final_solver', + })).toBe(false); + expect(shouldIncludeCompilerContextOverflow({ + workflow_mode: 'aggregator', + role_id: 'aggregator_validator', + })).toBe(false); + expect(shouldIncludeCompilerContextOverflow({})).toBe(false); + }); + + test('App suppresses duplicate overflow terminal entries but retains ordinary stops', () => { + expect(shouldRecordWorkflowStoppedActivity( + 'auto_research_stopped', + { reason: 'context_overflow' }, + )).toBe(false); + expect(shouldRecordWorkflowStoppedActivity( + 'leanoj_stopped', + { reason: 'context_overflow' }, + )).toBe(false); + expect(shouldRecordWorkflowStoppedActivity( + 'auto_research_stopped', + { reason: 'user_stop' }, + )).toBe(true); + }); + + test('App reformats persisted direct and terminal overflow records with model identity', () => { + localStorage.setItem('activity', JSON.stringify([ + { + event: 'context_overflow_error', + message: 'stale message', + data: { + message: 'Research stopped.', + configured_model: 'configured-model', + configured_provider: 'openrouter', + }, + }, + { + event: 'auto_research_stopped', + message: 'stale terminal message', + data: { + reason: 'context_overflow', + message: 'Research stopped.', + effective_model: 'fallback-model', + effective_provider: 'lm_studio', + }, + }, + ])); + + const restored = readPersistedLiveActivity('activity'); + expect(restored[0].message).toContain('Configured route: configured-model via openrouter'); + expect(restored[1].message).toContain('Route: fallback-model via lm_studio'); + }); + + test('App sanitizes credentials in legacy persisted activity records', () => { + localStorage.setItem('activity', JSON.stringify([{ + event: 'oauth_provider_failure', + message: 'Authorization: Bearer legacy-token', + data: { + provider: 'openai_codex_oauth', + access_token: 'legacy-access-token', + error_summary: 'callback https://example.test/?code=legacy-code&state=keep', + }, + }])); + + const [restored] = readPersistedLiveActivity('activity'); + expect(restored.message).not.toContain('legacy-token'); + expect(restored.data.access_token).toBe('[redacted]'); + expect(restored.data.error_summary).not.toContain('legacy-code'); + expect(restored.data.error_summary).toContain('state=keep'); + expect(restored.data.provider).toBe('openai_codex_oauth'); + }); + + test('App compacts durable activity without discarding sanitized messages', () => { + const compacted = compactLiveActivityEvent({ + event: 'proof_context_overflow', + message: 'Proof candidate exceeded its direct-context budget; Authorization: Bearer activity-secret', + data: { + workflow_mode: 'autonomous', + access_token: 'nested-secret', + error_summary: 'callback https://example.test/?code=oauth-code&state=keep', + }, + }); + + expect(compacted.message).toContain('Proof candidate exceeded its direct-context budget'); + expect(compacted.message).not.toContain('activity-secret'); + expect(compacted.data.access_token).toBe('[redacted]'); + expect(compacted.data.error_summary).not.toContain('oauth-code'); + expect(compacted.data.error_summary).toContain('state=keep'); + }); + + test('provider notification dismissals preserve legacy IDs and intermediate fingerprints', async () => { + localStorage.setItem( + 'dismissedOAuthProviderNotifications', + JSON.stringify(['legacy-notification']), + ); + expect(await isProviderNotificationDismissed('legacy-notification')).toBe(true); + + const fingerprint = await crypto.subtle.digest( + 'SHA-256', + new TextEncoder().encode('fingerprinted-notification'), + ); + const fingerprintHex = Array.from(new Uint8Array(fingerprint), byte => ( + byte.toString(16).padStart(2, '0') + )).join(''); + localStorage.setItem( + 'dismissedOAuthProviderNotifications', + JSON.stringify([fingerprintHex]), + ); + expect(await isProviderNotificationDismissed('fingerprinted-notification')).toBe(true); + }); + + test('provider notification dismissals use independent markers without lost updates', async () => { + await Promise.all([ + persistDismissedProviderNotificationId('notification-one'), + persistDismissedProviderNotificationId('notification-two'), + ]); + + expect(await isProviderNotificationDismissed('notification-one')).toBe(true); + expect(await isProviderNotificationDismissed('notification-two')).toBe(true); + expect( + Object.keys(localStorage).filter(key => ( + key.startsWith('dismissedOAuthProviderNotificationFingerprint:') + )), + ).toHaveLength(2); + }); + + test('Aggregator persisted overflow display includes stored model and provider', () => { + expect(formatAggregatorPersistedOverflowMessage({ + type: 'context_overflow_error', + message: 'legacy text without identity', + metadata: { + message: 'Research stopped.', + configured_model: 'aggregator-model', + configured_provider: 'openrouter', + }, + })).toBe('Research stopped. Configured route: aggregator-model via openrouter.'); + }); + + test('manual proof overflow routing assigns each source to exactly one manual feed', () => { + const aggregator = { source_type: 'brainstorm', source_id: 'manual_aggregator' }; + const compiler = { source_type: 'paper', source_id: 'manual_compiler_current' }; + const autonomous = { source_type: 'paper', source_id: 'paper_123', workflow_mode: 'autonomous' }; + const leanoj = { source_type: 'leanoj_final', source_id: 'leanoj_123' }; + + expect([ + shouldIncludeAggregatorProofContextOverflow(aggregator), + shouldIncludeCompilerProofContextOverflow(aggregator), + ]).toEqual([true, false]); + expect([ + shouldIncludeAggregatorProofContextOverflow(compiler), + shouldIncludeCompilerProofContextOverflow(compiler), + ]).toEqual([false, true]); + expect(shouldIncludeAggregatorProofContextOverflow(autonomous)).toBe(false); + expect(shouldIncludeCompilerProofContextOverflow(autonomous)).toBe(false); + expect(shouldIncludeAggregatorProofContextOverflow(leanoj)).toBe(false); + expect(shouldIncludeCompilerProofContextOverflow(leanoj)).toBe(false); + }); + + test('formatter distinguishes changed routes and tolerates partial route metadata', () => { + expect(formatContextOverflowActivityMessage({ + message: 'Proof context overflow.', + configured_model: 'primary-model', + configured_provider: 'openrouter', + effective_model: 'fallback-model', + effective_provider: 'lm_studio', + effective_host_provider: 'local-sibling', + })).toContain( + 'Effective route: fallback-model via lm_studio, host local-sibling. ' + + 'Configured route: primary-model via openrouter.' + ); + expect(formatContextOverflowActivityMessage({ + message: 'Proof context overflow.', + effective_provider: 'openrouter', + effective_host_provider: 'anthropic', + })).toContain('Route: openrouter, host anthropic.'); + }); + + test('compiler persistence compacts and bounds large event payloads', () => { + const compacted = compactCompilerActivityEvents(Array.from({ length: 2100 }, (_, index) => ({ + type: 'proof_context_overflow', + timestamp: `${index}`, + fullTimestamp: `2026-07-13T00:00:${index}.000Z`, + data: { + configured_model: 'configured-model', + effective_model: 'effective-model', + effective_host_provider: 'anthropic', + error_output: 'x'.repeat(5000), + }, + }))); + expect(compacted).toHaveLength(2000); + expect(compacted[0].data.configured_model).toBe('configured-model'); + expect(compacted[0].data.effective_host_provider).toBe('anthropic'); + expect(compacted[0].data.error_output.length).toBeLessThan(1300); + }); + + test('mounted manual logs route proof overflow to exactly one owning feed', async () => { + const compilerMetrics = { + construction: { acceptances: 0, rejections: 0, declines: 0, acceptance_rate: 0 }, + rigor: { acceptances: 0, rejections: 0, declines: 0, acceptance_rate: 0 }, + outline: { acceptances: 0, rejections: 0, declines: 0 }, + review: { acceptances: 0, rejections: 0, declines: 0 }, + minuscule_edit_count: 0, + paper_word_count: 0, + total_submissions: 0, + }; + vi.spyOn(api, 'getStatus').mockResolvedValue({ + queue_size: 0, + total_acceptances: 0, + total_rejections: 0, + submitter_statuses: [], + }); + vi.spyOn(compilerAPI, 'getMetrics').mockResolvedValue({ data: compilerMetrics }); + vi.spyOn(compilerAPI, 'getStatus').mockResolvedValue({ + data: { current_mode: 'construction', is_running: true }, + }); + vi.spyOn(globalThis, 'fetch').mockResolvedValue({ + ok: true, + json: async () => ({ events: [] }), + }); + + render( + <> + <AggregatorLogs /> + <CompilerLogs /> + </> + ); + + await waitFor(() => { + expect(api.getStatus).toHaveBeenCalled(); + expect(compilerAPI.getStatus).toHaveBeenCalled(); + expect(globalThis.fetch).toHaveBeenCalled(); + }); + await act(async () => { + await new Promise((resolve) => setTimeout(resolve, 0)); + websocket.emit('proof_context_overflow', { + source_type: 'brainstorm', + source_id: 'manual_aggregator', + message: 'Aggregator-only overflow.', + configured_model: 'agg-model', + }); + websocket.emit('proof_context_overflow', { + source_type: 'paper', + source_id: 'manual_compiler_current', + message: 'Compiler-only overflow.', + configured_model: 'compiler-model', + }); + websocket.emit('proof_context_overflow', { + source_type: 'paper', + source_id: 'autonomous-paper', + workflow_mode: 'autonomous', + message: 'App-owned overflow.', + }); + }); + + await waitFor(() => { + expect(screen.getAllByText(/Aggregator-only overflow/)).toHaveLength(1); + expect(screen.getAllByText(/Compiler-only overflow/)).toHaveLength(1); + }); + expect(screen.queryByText(/App-owned overflow/)).toBeNull(); + }); +}); diff --git a/frontend/src/components/leanoj/LeanOJMathematicalProofs.jsx b/frontend/src/components/leanoj/LeanOJMathematicalProofs.jsx index a77f667..c409ed8 100644 --- a/frontend/src/components/leanoj/LeanOJMathematicalProofs.jsx +++ b/frontend/src/components/leanoj/LeanOJMathematicalProofs.jsx @@ -47,6 +47,20 @@ function formatSolverName(solver) { return String(solver || 'Proof Solver').replace(/^LeanOJ\b/, 'Proof Solver'); } +function getCurrentProofDisplay(proof) { + const prompt = String(proof.user_prompt || '').trim(); + const statement = String(proof.theorem_statement || '').trim(); + const sourceTitle = String(proof.source_title || '').trim(); + const statementIsPrompt = Boolean(prompt && statement === prompt); + const sourceIsPrompt = Boolean(prompt && sourceTitle === prompt); + return { + title: statementIsPrompt + ? (proof.theorem_name || (proof.proof_kind === 'final' ? 'Final verified submission' : 'Verified proof')) + : (statement || proof.theorem_name || proof.proof_id), + summary: sourceIsPrompt ? '' : sourceTitle, + }; +} + export default function LeanOJMathematicalProofs({ api, status, refreshToken = 0 }) { const [proofs, setProofs] = useState([]); const [loading, setLoading] = useState(true); @@ -106,7 +120,6 @@ export default function LeanOJMathematicalProofs({ api, status, refreshToken = 0 } return proofs; }, [filter, proofs]); - const handleDownloadLean = (proof) => { if (!proof.lean_code) return; downloadTextFile(proof.lean_code, `${proof.theorem_name || proof.proof_id}.lean`); @@ -210,6 +223,7 @@ export default function LeanOJMathematicalProofs({ api, status, refreshToken = 0 {visibleProofs.map((proof) => { const isExpanded = expandedProofId === proof.library_id; const badge = getProofBadge(proof); + const display = getCurrentProofDisplay(proof); return ( <article key={proof.library_id} className={`math-proof-card ${badge.cardClass}`}> <div className="math-proof-card-header"> @@ -222,9 +236,9 @@ export default function LeanOJMathematicalProofs({ api, status, refreshToken = 0 {proof.session_id} </span> </div> - <h3>{proof.theorem_statement}</h3> + <h3>{display.title}</h3> <p className="math-proof-summary"> - {truncate(proof.source_title || proof.user_prompt || 'Lean 4 verified this Proof Solver proof.')} + {truncate(display.summary || 'Lean 4 verified this Proof Solver proof.')} </p> </div> diff --git a/frontend/src/components/leanoj/LeanOJMathematicalProofs.test.jsx b/frontend/src/components/leanoj/LeanOJMathematicalProofs.test.jsx new file mode 100644 index 0000000..09e5550 --- /dev/null +++ b/frontend/src/components/leanoj/LeanOJMathematicalProofs.test.jsx @@ -0,0 +1,51 @@ +import React from 'react'; +import { render, screen } from '@testing-library/react'; +import LeanOJMathematicalProofs from './LeanOJMathematicalProofs'; + +test('shows current-run proofs directly without rendering the user prompt', async () => { + const proof = { + library_id: 'run-1:proof-1', + proof_id: 'proof-1', + theorem_statement: 'theorem current_run : True', + user_prompt: 'A very long private current-run prompt that must not be displayed', + source_title: '', + lean_code: 'theorem current_run : True := by trivial', + proof_kind: 'subproof', + novelty_tier: 'not_novel', + novel: false, + }; + const api = { + getProofs: vi.fn().mockResolvedValue({ proofs: [proof] }), + }; + + render(<LeanOJMathematicalProofs api={api} status={{ session_id: 'run-1' }} />); + + expect(await screen.findByText(proof.theorem_statement)).toBeInTheDocument(); + expect(screen.queryByText(proof.user_prompt)).not.toBeInTheDocument(); + expect(screen.queryByRole('button', { name: new RegExp(proof.user_prompt, 'i') })).not.toBeInTheDocument(); +}); + +test('does not render the run prompt when a final proof record uses it as statement and source', async () => { + const prompt = `Solve this run without displaying the prompt ${'x'.repeat(500)}`; + const proof = { + library_id: 'run-2:final', + proof_id: 'final', + theorem_name: 'final_solution', + theorem_statement: prompt, + user_prompt: prompt, + source_title: prompt, + lean_code: 'theorem final_solution : True := by trivial', + proof_kind: 'final', + novelty_tier: 'not_novel', + novel: false, + }; + const api = { + getProofs: vi.fn().mockResolvedValue({ proofs: [proof] }), + }; + + render(<LeanOJMathematicalProofs api={api} status={{ session_id: 'run-2', user_prompt: prompt }} />); + + expect(await screen.findByText('final_solution')).toBeInTheDocument(); + expect(screen.queryByText(prompt)).not.toBeInTheDocument(); + expect(screen.getByText('Lean 4 verified this Proof Solver proof.')).toBeInTheDocument(); +}); diff --git a/frontend/src/components/leanoj/LeanOJProofLibrary.jsx b/frontend/src/components/leanoj/LeanOJProofLibrary.jsx index a2b3eb0..142df8d 100644 --- a/frontend/src/components/leanoj/LeanOJProofLibrary.jsx +++ b/frontend/src/components/leanoj/LeanOJProofLibrary.jsx @@ -1,5 +1,11 @@ -import React, { useEffect, useMemo, useState } from 'react'; +import React, { useEffect, useMemo, useRef, useState } from 'react'; import { downloadTextFile } from '../../utils/downloadHelpers'; +import { + getCanonicalProofIdentity, + getLeanOJProofPresentation, + sanitizeDomId, +} from '../../utils/proofPresentation'; +import { formatRunPromptPreview } from '../../utils/researchRunHistory'; import '../autonomous/FinalAnswerLibrary.css'; import '../autonomous/ProofLibrary.css'; @@ -17,36 +23,68 @@ function truncate(text, maxLength = 260) { return text.length > maxLength ? `${text.slice(0, maxLength)}...` : text; } -function getProofBadge(proof) { - if (proof.proof_kind === 'final') { - return { cssClass: 'proof-badge--gold', cardClass: 'proof-card--gold', label: 'Final Verified Submission' }; - } - return { cssClass: 'proof-badge--silver', cardClass: 'proof-card--silver', label: 'Verified Proof Fragment' }; -} - function formatSolverName(solver) { return String(solver || 'Proof Solver').replace(/^LeanOJ\b/, 'Proof Solver'); } -export default function LeanOJProofLibrary({ api, refreshToken = 0 }) { +function getLeanOJProofCardId(proof) { + return getCanonicalProofIdentity({ ...proof, corpus: proof.corpus || 'leanoj' }); +} + +function getLeanOJRunKey(proof) { + return proof.run_id || proof.session_id || `orphan:${proof.proof_id}`; +} + +function matchesSelectedLeanOJProof(proof, selectedProofId, selectedSessionId = '', selectedRunId = '') { + if (!selectedProofId) return false; + const proofIds = [ + proof.proof_id, + proof.library_id, + proof.search_id, + proof.lean_code_hash, + proof.theorem_statement_hash, + ].filter(Boolean).map(String); + if (!proofIds.includes(String(selectedProofId))) { + return false; + } + if (selectedRunId) return getLeanOJRunKey(proof) === selectedRunId; + return !selectedSessionId || !proof.session_id || proof.session_id === selectedSessionId; +} + +export default function LeanOJProofLibrary({ + api, + refreshToken = 0, + selectedProofId = null, + selectedSessionId = '', + selectedRunId = '', +}) { const [proofs, setProofs] = useState([]); const [sessions, setSessions] = useState([]); const [loading, setLoading] = useState(true); const [error, setError] = useState(''); const [expandedId, setExpandedId] = useState(null); const [expandedProof, setExpandedProof] = useState(null); + const [expandedSessions, setExpandedSessions] = useState(() => new Set()); const [loadingContentId, setLoadingContentId] = useState(null); const [searchTerm, setSearchTerm] = useState(''); const [filterKind, setFilterKind] = useState('all'); + const detailGenerationRef = useRef(0); + const libraryGenerationRef = useRef(0); const loadProofLibrary = async () => { + const generation = ++libraryGenerationRef.current; + detailGenerationRef.current += 1; + setExpandedId(null); + setExpandedProof(null); try { setLoading(true); setError(''); const response = await api.getProofLibrary(true); + if (generation !== libraryGenerationRef.current) return; setProofs(response.proofs || []); setSessions(response.sessions || []); } catch (err) { + if (generation !== libraryGenerationRef.current) return; if (err.status === 404) { setProofs([]); setSessions([]); @@ -55,7 +93,7 @@ export default function LeanOJProofLibrary({ api, refreshToken = 0 }) { } setError(err.message || 'Failed to load Proof Solver proof works library'); } finally { - setLoading(false); + if (generation === libraryGenerationRef.current) setLoading(false); } }; @@ -63,6 +101,11 @@ export default function LeanOJProofLibrary({ api, refreshToken = 0 }) { loadProofLibrary(); }, [refreshToken]); + useEffect(() => () => { + libraryGenerationRef.current += 1; + detailGenerationRef.current += 1; + }, []); + const filteredProofs = useMemo(() => { const lowerSearch = searchTerm.trim().toLowerCase(); return proofs.filter((proof) => { @@ -74,6 +117,7 @@ export default function LeanOJProofLibrary({ api, refreshToken = 0 }) { (proof.theorem_statement || '').toLowerCase().includes(lowerSearch) || (proof.source_title || '').toLowerCase().includes(lowerSearch) || (proof.user_prompt || '').toLowerCase().includes(lowerSearch) || + (proof.run_id || '').toLowerCase().includes(lowerSearch) || (proof.session_id || '').toLowerCase().includes(lowerSearch) ); }); @@ -82,7 +126,7 @@ export default function LeanOJProofLibrary({ api, refreshToken = 0 }) { const proofsBySession = useMemo(() => { const map = new Map(); for (const proof of filteredProofs) { - const sessionId = proof.session_id || 'unknown'; + const sessionId = getLeanOJRunKey(proof); if (!map.has(sessionId)) map.set(sessionId, []); map.get(sessionId).push(proof); } @@ -90,7 +134,22 @@ export default function LeanOJProofLibrary({ api, refreshToken = 0 }) { }, [filteredProofs]); const visibleSessions = useMemo(() => { - return sessions.filter((session) => proofsBySession.has(session.session_id)); + const metadata = new Map(); + sessions.forEach((session) => { + if (session.run_id) metadata.set(session.run_id, session); + if (session.session_id && !metadata.has(session.session_id)) metadata.set(session.session_id, session); + }); + return Array.from(proofsBySession.keys()).map((runKey) => { + const matched = metadata.get(runKey); + return matched ? { ...matched, group_key: runKey } : { + session_id: proofsBySession.get(runKey)?.[0]?.session_id || '', + run_id: proofsBySession.get(runKey)?.[0]?.run_id || '', + group_key: runKey, + user_prompt: proofsBySession.get(runKey)?.[0]?.user_prompt || 'Legacy Proof Solver run', + updated_at: proofsBySession.get(runKey)?.[0]?.created_at, + phase: 'legacy', + }; + }); }, [proofsBySession, sessions]); const counts = useMemo(() => ({ @@ -100,8 +159,9 @@ export default function LeanOJProofLibrary({ api, refreshToken = 0 }) { }), [proofs]); const handleExpand = async (proof) => { - const id = proof.library_id || `${proof.session_id}:${proof.proof_id}`; + const id = getLeanOJProofCardId(proof); if (expandedId === id) { + detailGenerationRef.current += 1; setExpandedId(null); setExpandedProof(null); return; @@ -109,16 +169,75 @@ export default function LeanOJProofLibrary({ api, refreshToken = 0 }) { setExpandedId(id); setLoadingContentId(id); + const generation = ++detailGenerationRef.current; try { const fullProof = await api.getLibraryProof(proof.session_id, proof.proof_id); - setExpandedProof(fullProof); + if (generation === detailGenerationRef.current) setExpandedProof(fullProof); } catch { - setExpandedProof(proof); + if (generation === detailGenerationRef.current) setExpandedProof(proof); } finally { - setLoadingContentId(null); + if (generation === detailGenerationRef.current) setLoadingContentId(null); } }; + useEffect(() => { + if (!selectedProofId || loading) return; + const visibleMatch = filteredProofs.find((proof) => matchesSelectedLeanOJProof(proof, selectedProofId, selectedSessionId, selectedRunId)); + if (visibleMatch) return; + const searchHiddenMatch = proofs.find((proof) => matchesSelectedLeanOJProof(proof, selectedProofId, selectedSessionId, selectedRunId)); + if (searchHiddenMatch && searchTerm) { + setSearchTerm(''); + return; + } + const hiddenKindMatch = proofs.find((proof) => matchesSelectedLeanOJProof(proof, selectedProofId, selectedSessionId, selectedRunId)); + if (hiddenKindMatch && filterKind !== 'all') { + setFilterKind('all'); + } + }, [filterKind, filteredProofs, loading, proofs, searchTerm, selectedProofId, selectedRunId, selectedSessionId]); + + useEffect(() => { + if (!selectedProofId || loading) return; + const match = filteredProofs.find((proof) => matchesSelectedLeanOJProof(proof, selectedProofId, selectedSessionId, selectedRunId)); + if (!match) return; + const id = getLeanOJProofCardId(match); + const sessionKey = getLeanOJRunKey(match); + setExpandedSessions((previous) => new Set(previous).add(sessionKey)); + setExpandedId(id); + setLoadingContentId(id); + const generation = ++detailGenerationRef.current; + let cancelled = false; + api.getLibraryProof(match.session_id, match.proof_id) + .then((fullProof) => { + if (!cancelled && generation === detailGenerationRef.current) setExpandedProof(fullProof); + }) + .catch(() => { + if (!cancelled && generation === detailGenerationRef.current) setExpandedProof(match); + }) + .finally(() => { + if (!cancelled) { + setLoadingContentId(null); + setTimeout(() => { + document.getElementById(sanitizeDomId(id, 'leanoj-proof-card'))?.scrollIntoView({ behavior: 'smooth', block: 'center' }); + }, 0); + } + }); + return () => { + cancelled = true; + }; + }, [api, filteredProofs, loading, selectedProofId, selectedRunId, selectedSessionId]); + + const toggleSession = (sessionId) => { + setExpandedSessions((previous) => { + const next = new Set(previous); + if (next.has(sessionId)) next.delete(sessionId); + else next.add(sessionId); + return next; + }); + detailGenerationRef.current += 1; + setExpandedId(null); + setExpandedProof(null); + }; + const handleDownloadLean = async (proof, event) => { event?.stopPropagation(); @@ -178,6 +297,7 @@ export default function LeanOJProofLibrary({ api, refreshToken = 0 }) { <div className="library-controls"> <input + aria-label="Filter Proof Solver proof works" className="search-input" type="text" placeholder="Search by theorem, problem, session, or Proof Solver source..." @@ -185,13 +305,13 @@ export default function LeanOJProofLibrary({ api, refreshToken = 0 }) { onChange={(event) => setSearchTerm(event.target.value)} /> <div className="filter-buttons"> - <button className={filterKind === 'all' ? 'active' : ''} onClick={() => setFilterKind('all')}> + <button className={filterKind === 'all' ? 'active' : ''} aria-pressed={filterKind === 'all'} onClick={() => setFilterKind('all')}> All </button> - <button className={filterKind === 'final' ? 'active' : ''} onClick={() => setFilterKind('final')}> + <button className={filterKind === 'final' ? 'active' : ''} aria-pressed={filterKind === 'final'} onClick={() => setFilterKind('final')}> Final </button> - <button className={filterKind === 'subproof' ? 'active' : ''} onClick={() => setFilterKind('subproof')}> + <button className={filterKind === 'subproof' ? 'active' : ''} aria-pressed={filterKind === 'subproof'} onClick={() => setFilterKind('subproof')}> Proof Fragments </button> </div> @@ -210,12 +330,23 @@ export default function LeanOJProofLibrary({ api, refreshToken = 0 }) { ) : ( <div className="run-history-groups"> {visibleSessions.map((session) => { - const sessionProofs = proofsBySession.get(session.session_id) || []; + const groupKey = session.group_key || session.run_id || session.session_id; + const sessionProofs = proofsBySession.get(groupKey) || []; + const isSessionExpanded = expandedSessions.has(groupKey); + const sessionRegionId = sanitizeDomId(groupKey, 'leanoj-proof-session'); return ( - <div key={session.session_id} className="run-history-group"> - <div className="run-history-group-header"> + <div key={groupKey} className={`run-history-group proof-prompt-group ${isSessionExpanded ? 'expanded' : ''}`}> + <button + type="button" + className="run-history-group-header proof-prompt-group-header" + onClick={() => toggleSession(groupKey)} + aria-expanded={isSessionExpanded} + aria-controls={sessionRegionId} + > <div className="run-history-group-heading"> - <h3 className="run-history-group-title">{session.user_prompt}</h3> + <h3 className="run-history-group-title"> + {formatRunPromptPreview(session.user_prompt || sessionProofs[0]?.user_prompt || 'Legacy Proof Solver run')} + </h3> <p className="run-history-group-subtitle"> {sessionProofs.length} proof work{sessionProofs.length !== 1 ? 's' : ''} - {formatDate(session.updated_at)} </p> @@ -229,19 +360,25 @@ export default function LeanOJProofLibrary({ api, refreshToken = 0 }) { {session.phase && ( <span className="run-history-group-badge">{session.phase}</span> )} + <span className="proof-prompt-group-chevron" aria-hidden="true"> + {isSessionExpanded ? '▲' : '▼'} + </span> </div> - </div> + </button> - <div className="run-history-group-body"> + <div id={sessionRegionId} className="run-history-group-body" role="region" aria-label={`Proof works for ${formatRunPromptPreview(session.user_prompt || sessionProofs[0]?.user_prompt || groupKey)}`} hidden={!isSessionExpanded}> + {isSessionExpanded && ( <div className="answer-list"> {sessionProofs.map((proof) => { - const id = proof.library_id || `${proof.session_id}:${proof.proof_id}`; + const id = getLeanOJProofCardId(proof); + const cardDomId = sanitizeDomId(id, 'leanoj-proof-card'); + const detailsDomId = sanitizeDomId(id, 'leanoj-proof-details'); const isExpanded = expandedId === id; - const badge = getProofBadge(proof); + const badge = getLeanOJProofPresentation(proof); return ( - <div key={id} className={`answer-card proof-card ${isExpanded ? 'expanded' : ''} ${badge.cardClass}`}> - <div className="answer-header" onClick={() => handleExpand(proof)}> + <div id={cardDomId} key={id} className={`answer-card proof-card ${isExpanded ? 'expanded' : ''} ${badge.cardClass}`}> + <div className="answer-header"> <div className="answer-title-row"> <h4 className="answer-title proof-title"> {proof.theorem_name || proof.proof_id} @@ -254,7 +391,14 @@ export default function LeanOJProofLibrary({ api, refreshToken = 0 }) { > Download .lean </button> - <button className="expand-button"> + <button + type="button" + className="expand-button" + onClick={() => handleExpand(proof)} + aria-expanded={isExpanded} + aria-controls={detailsDomId} + aria-label={`${isExpanded ? 'Collapse' : 'Expand'} proof ${proof.theorem_name || proof.proof_id}`} + > {isExpanded ? 'Hide' : 'View'} </button> </div> @@ -289,9 +433,9 @@ export default function LeanOJProofLibrary({ api, refreshToken = 0 }) { </div> </div> - {isExpanded && ( - <div className="answer-content"> - {loadingContentId === id ? ( + <div id={detailsDomId} className="answer-content" role="region" aria-label={`Details for ${proof.theorem_name || proof.proof_id}`} hidden={!isExpanded}> + {isExpanded && ( + loadingContentId === id ? ( <div className="library-loading" style={{ padding: '20px' }}> <span className="library-loading__icon">↻</span> <span className="library-loading__text">Loading proof work details...</span> @@ -328,13 +472,14 @@ export default function LeanOJProofLibrary({ api, refreshToken = 0 }) { )} </div> </div> - ) : null} + ) : null + )} </div> - )} </div> ); })} </div> + )} </div> </div> ); diff --git a/frontend/src/components/leanoj/LeanOJProofLibrary.test.jsx b/frontend/src/components/leanoj/LeanOJProofLibrary.test.jsx new file mode 100644 index 0000000..5e9a3e6 --- /dev/null +++ b/frontend/src/components/leanoj/LeanOJProofLibrary.test.jsx @@ -0,0 +1,117 @@ +import React from 'react'; +import { render, screen, waitFor } from '@testing-library/react'; +import userEvent from '@testing-library/user-event'; +import LeanOJProofLibrary from './LeanOJProofLibrary'; + +vi.mock('../../utils/downloadHelpers', () => ({ + downloadTextFile: vi.fn(), +})); + +beforeEach(() => { + vi.clearAllMocks(); + Element.prototype.scrollIntoView = vi.fn(); +}); + +test('selected proof id expands and hydrates matching LeanOJ proof history card', async () => { + const api = { + getProofLibrary: vi.fn().mockResolvedValue({ + proofs: [ + { + proof_id: 'leanoj-proof-1', + session_id: 'leanoj-session-1', + theorem_name: 'LeanOJ.Helper', + theorem_statement: 'theorem leanoj_helper : True', + proof_kind: 'subproof', + source_title: 'LeanOJ source', + created_at: '2026-06-12T00:00:00+00:00', + }, + ], + sessions: [ + { + session_id: 'leanoj-session-1', + user_prompt: 'LeanOJ run', + updated_at: '2026-06-12T00:00:00+00:00', + }, + ], + }), + getLibraryProof: vi.fn().mockResolvedValue({ + proof_id: 'leanoj-proof-1', + session_id: 'leanoj-session-1', + theorem_name: 'LeanOJ.Helper', + theorem_statement: 'Hydrated LeanOJ proof statement.', + lean_code: 'theorem leanoj_helper : True := by trivial', + }), + }; + + render( + <LeanOJProofLibrary + api={api} + selectedProofId="leanoj-proof-1" + selectedSessionId="leanoj-session-1" + /> + ); + + expect(await screen.findByText('LeanOJ.Helper')).toBeInTheDocument(); + await waitFor(() => { + expect(api.getLibraryProof).toHaveBeenCalledWith('leanoj-session-1', 'leanoj-proof-1'); + }); + expect(await screen.findByText('Hydrated LeanOJ proof statement.')).toBeInTheDocument(); + expect(Element.prototype.scrollIntoView).toHaveBeenCalled(); +}); + +test('keeps LeanOJ session proofs collapsed until the prompt is expanded', async () => { + const user = userEvent.setup(); + const api = { + getProofLibrary: vi.fn().mockResolvedValue({ + proofs: [{ + proof_id: 'leanoj-proof-2', + session_id: 'leanoj-session-2', + theorem_name: 'LeanOJ.Collapsed', + theorem_statement: 'theorem collapsed : True', + proof_kind: 'final', + }], + sessions: [{ + session_id: 'leanoj-session-2', + user_prompt: 'Solve the LeanOJ prompt', + updated_at: '2026-06-12T00:00:00+00:00', + }], + }), + getLibraryProof: vi.fn(), + }; + + render(<LeanOJProofLibrary api={api} />); + + const groupButton = await screen.findByRole('button', { name: /Solve the LeanOJ prompt/i }); + expect(groupButton).toHaveAttribute('aria-expanded', 'false'); + expect(screen.queryByText('LeanOJ.Collapsed')).not.toBeInTheDocument(); + await user.click(groupButton); + expect(screen.getByText('LeanOJ.Collapsed')).toBeInTheDocument(); +}); + +test('bounds long prompt text in historical group headers', async () => { + const longPrompt = `Solve ${'a'.repeat(1000)}`; + const api = { + getProofLibrary: vi.fn().mockResolvedValue({ + proofs: [{ + proof_id: 'leanoj-proof-3', + session_id: 'leanoj-session-3', + theorem_name: 'LeanOJ.LongPrompt', + theorem_statement: 'theorem long_prompt : True', + proof_kind: 'final', + }], + sessions: [{ + session_id: 'leanoj-session-3', + user_prompt: longPrompt, + updated_at: '2026-06-12T00:00:00+00:00', + }], + }), + getLibraryProof: vi.fn(), + }; + + render(<LeanOJProofLibrary api={api} />); + + const groupButton = await screen.findByRole('button', { name: /Solve a+/i }); + expect(groupButton).toHaveTextContent('...'); + expect(groupButton).not.toHaveTextContent(longPrompt); +}); + diff --git a/frontend/src/components/leanoj/LeanOJSettings.jsx b/frontend/src/components/leanoj/LeanOJSettings.jsx index 1f07abd..2c2bcbc 100644 --- a/frontend/src/components/leanoj/LeanOJSettings.jsx +++ b/frontend/src/components/leanoj/LeanOJSettings.jsx @@ -36,6 +36,7 @@ import { } from '../../utils/leanojProfiles'; import HelpTooltip from '../HelpTooltip'; import HighlightedModelsSidebar from '../HighlightedModelsSidebar'; +import OpenRouterFreeModelsControl from '../OpenRouterFreeModelsControl'; import RawSettingsEditor from '../RawSettingsEditor'; import '../settings-common.css'; @@ -270,7 +271,7 @@ function RoleEditor(props) { <h4>{title}</h4> {title === 'Assistant' && ( <p className="settings-info"> - Runs in parallel during topic, brainstorm, path, master-proof edit, and final proof work to retrieve up to 7 relevant memory supports from Session History Memory and SyntheticLib4 when enabled. Validators never receive Assistant context. + Runs in parallel during topic, brainstorm, path, master-proof edit, and final proof work to retrieve up to 7 relevant verified proof-memory supports from Session History Memory and SyntheticLib4 when enabled. Validators never receive Assistant context. </p> )} {disabled && ( @@ -1005,18 +1006,16 @@ export default function LeanOJSettings({ > 🔗 OpenRouter Model List </button> - <label className="settings-checkbox-label model-refresh-controls__toggle" style={{ cursor: isRunning ? 'not-allowed' : 'pointer' }}> - <input - type="checkbox" - checked={settings.freeOnly} - onChange={(event) => updateSettings({ freeOnly: event.target.checked })} - disabled={isRunning} - /> - Free models only - </label> + <OpenRouterFreeModelsControl + checked={settings.freeOnly} + disabled={isRunning} + onChange={(freeOnly) => updateSettings({ freeOnly })} + /> </> )} - {developerModeEnabled ? ( + {developerModeEnabled && ( + <> + {hasOpenRouterKey && <span className="model-refresh-controls__divider" aria-hidden="true" />} <label className="settings-checkbox-label model-refresh-controls__toggle" style={{ cursor: isRunning ? 'not-allowed' : 'pointer' }}> <input type="checkbox" @@ -1026,10 +1025,7 @@ export default function LeanOJSettings({ /> Edit Raw </label> - ) : ( - <span className="settings-developer-mode-hint"> - Developer mode: press Shift + Z + X to toggle raw JSON settings. - </span> + </> )} </div> diff --git a/frontend/src/components/settings-common.css b/frontend/src/components/settings-common.css index 8145f18..cce1746 100644 --- a/frontend/src/components/settings-common.css +++ b/frontend/src/components/settings-common.css @@ -458,12 +458,19 @@ /* Dark text input */ .input-dark { - padding: 0.5rem; - background: var(--surface-3); - border: 1px solid var(--border-default); + padding: 0.55rem 0.65rem; + background: #22222c; + border: 1px solid rgba(255, 255, 255, 0.18); border-radius: var(--radius-sm); - color: var(--text-primary); + color: #f4f5ff; width: 100%; + box-sizing: border-box; + caret-color: #f4f5ff; +} + +.input-dark::placeholder { + color: rgba(244, 245, 255, 0.52); + opacity: 1; } .input-dark:focus { @@ -472,6 +479,13 @@ box-shadow: 0 0 0 3px var(--gold-glow); } +.input-dark:disabled { + background: #1b1b24; + color: rgba(244, 245, 255, 0.55); + cursor: not-allowed; + opacity: 0.75; +} + /* Dark textarea (monospace) */ .textarea-dark-mono { width: 100%; @@ -866,20 +880,111 @@ display: flex; flex-wrap: wrap; align-items: center; - gap: 0.5rem; + gap: 0.6rem; margin-bottom: 1rem; } .model-refresh-controls__toggle { + display: inline-flex; + align-items: center; + min-height: 2rem; white-space: nowrap; font-size: 0.85rem; gap: 0.35rem; + line-height: 1; } .model-refresh-controls__toggle input[type="checkbox"] { margin-right: 0.35rem !important; } +.model-refresh-controls__divider { + align-self: stretch; + width: 1px; + min-height: 1.5rem; + background: var(--border-subtle); +} + +.settings-developer-mode-hint.model-refresh-controls__hint { + flex-basis: 100%; + margin-top: 0.1rem; +} + +.openrouter-free-models-control { + display: inline-flex; + flex-direction: column; + align-items: flex-start; + gap: 0.25rem; + min-height: 2.8rem; + line-height: 1; + white-space: nowrap; +} + +.openrouter-free-models-control__checkbox { + display: inline-grid; + grid-template-columns: auto auto; + align-items: center; + column-gap: 0.45rem; + width: max-content; + margin: 0; + color: var(--text-primary); + font-size: 0.85rem; + font-weight: 500; + min-height: 1.25rem; + line-height: 1; + cursor: pointer; +} + +.openrouter-free-models-control__checkbox input[type="checkbox"] { + flex: 0 0 auto; + width: auto; + min-width: 0; + height: auto; + margin-top: 0; + margin-right: 0; + padding: 0; + cursor: pointer; +} + +.openrouter-free-models-control__help { + display: inline-grid; + grid-template-columns: auto auto; + align-items: center; + column-gap: 0.25rem; + padding-left: 1.35rem; + min-height: 1.25rem; + color: var(--text-secondary); + font-size: 0.85rem; + line-height: 1; + white-space: nowrap; +} + +.openrouter-free-models-control__help-label { + color: var(--text-primary); + font-weight: 500; + line-height: 1; +} + +.help-tooltip-btn.openrouter-free-models-control__help-button { + width: 18px; + height: 18px; + min-width: 18px; + min-height: 18px; + padding: 0; + line-height: 1; + transform: none; +} + +.help-tooltip-btn.openrouter-free-models-control__help-button:hover, +.help-tooltip-btn.openrouter-free-models-control__help-button:focus-visible { + transform: none; +} + +.openrouter-free-models-control__tooltip { + width: min(340px, calc(100vw - 2rem)); + white-space: normal; +} + /* Flex column for checkbox groups */ .checkbox-group-col { display: flex; diff --git a/frontend/src/hooks/useProofCheckRuntime.js b/frontend/src/hooks/useProofCheckRuntime.js index 11fdd34..56beae8 100644 --- a/frontend/src/hooks/useProofCheckRuntime.js +++ b/frontend/src/hooks/useProofCheckRuntime.js @@ -298,7 +298,7 @@ export function useProofCheckRuntime() { ...prev, [sourceKey]: { status: 'running', - candidateCount: prev[sourceKey]?.candidateCount ?? null, + candidateCount: null, }, })); setQueuedChecks((prev) => { @@ -322,6 +322,17 @@ export function useProofCheckRuntime() { })); }); + const unsubscribeNoCandidates = websocket.on('proof_check_no_candidates', (data) => { + const sourceKey = buildSourceKey(data.source_type, data.source_id); + setActiveChecks((prev) => ({ + ...prev, + [sourceKey]: { + status: 'running', + candidateCount: 0, + }, + })); + }); + const unsubscribeComplete = websocket.on('proof_check_complete', (data) => { const sourceKey = buildSourceKey(data.source_type, data.source_id); setActiveChecks((prev) => { @@ -346,6 +357,7 @@ export function useProofCheckRuntime() { return () => { unsubscribeStarted(); unsubscribeCandidates(); + unsubscribeNoCandidates(); unsubscribeComplete(); }; }, [refreshProofStatus]); diff --git a/frontend/src/index.css b/frontend/src/index.css index 968e26c..7bb9ad8 100644 --- a/frontend/src/index.css +++ b/frontend/src/index.css @@ -104,21 +104,33 @@ code { } .app { + --workflow-panel-width: 320px; + --workflow-panel-gap: 30px; + --workflow-panel-safe-area: calc(var(--workflow-panel-width) + var(--workflow-panel-gap)); min-height: 100vh; display: flex; flex-direction: column; position: relative; } +.app.workflow-panel-collapsed { + --workflow-panel-width: 50px; + --workflow-panel-gap: 10px; +} + /* ============================================ Banner & Logo Section ============================================ */ .app-banner { - background: linear-gradient(160deg, var(--surface-1) 0%, var(--surface-3) 50%, var(--surface-1) 100%); - border-bottom: 1px solid var(--gold-dim); - box-shadow: 0 2px 20px rgba(30, 255, 28, 0.08); - padding: 1.5rem 2rem; - text-align: center; + background: + radial-gradient(circle at 18% 15%, rgba(212, 175, 55, 0.14), transparent 34%), + radial-gradient(circle at 82% 0%, rgba(30, 255, 28, 0.08), transparent 30%), + linear-gradient(135deg, rgba(8, 10, 13, 0.98) 0%, rgba(17, 20, 24, 0.96) 48%, rgba(8, 10, 13, 0.98) 100%); + border-bottom: 1px solid rgba(212, 175, 55, 0.28); + box-shadow: + 0 1px 0 rgba(255, 255, 255, 0.04) inset, + 0 14px 36px rgba(0, 0, 0, 0.32); + padding: 0.6rem 2rem; position: relative; overflow: hidden; isolation: isolate; @@ -135,12 +147,12 @@ code { background: linear-gradient( 90deg, transparent 0%, - rgba(30, 255, 28, 0.03) 25%, - rgba(30, 255, 28, 0.06) 50%, - rgba(30, 255, 28, 0.03) 75%, + rgba(212, 175, 55, 0.025) 25%, + rgba(255, 255, 255, 0.035) 50%, + rgba(30, 255, 28, 0.025) 75%, transparent 100% ); - animation: banner-shimmer 8s ease-in-out infinite; + animation: banner-shimmer 12s ease-in-out infinite; pointer-events: none; } @@ -160,68 +172,121 @@ code { } .banner-content { + align-items: center; + display: flex; + flex-direction: column; + gap: 0.38rem; + justify-content: center; + margin: 0 auto; + max-width: 1180px; position: relative; + text-align: center; z-index: 1; transform: translateZ(0); backface-visibility: hidden; } -.banner-title { - font-size: 1.75rem; - font-weight: 700; - margin: 0 0 0.35rem 0; - letter-spacing: 0.5px; +.banner-brand { + align-items: center; display: flex; + flex-direction: row; + gap: 1rem; + justify-content: center; + min-width: 0; +} + +.banner-logo-link { align-items: center; + display: inline-flex; + flex: 0 0 auto; justify-content: center; - gap: 0.75rem; - flex-wrap: wrap; + line-height: 0; + transition: transform 0.18s ease; + user-select: none; + -webkit-user-select: none; +} + +.banner-logo-link:hover { + transform: translateY(-1px); +} + +.banner-logo-link:focus-visible { + outline: 2px solid rgba(30, 255, 28, 0.75); + outline-offset: 4px; +} + +.banner-logo { + display: block; + filter: + brightness(1.55) + contrast(1.22) + drop-shadow(0 0 12px rgba(255, 255, 255, 0.32)) + drop-shadow(0 10px 20px rgba(0, 0, 0, 0.55)); + height: auto; + object-fit: contain; + width: clamp(68px, 6vw, 92px); +} + +.banner-identity { + align-items: flex-start; + display: flex; + flex-direction: column; + min-width: 0; + text-align: left; +} + +.banner-title { + align-items: flex-start; + color: #f7f8f4; + display: flex; + flex-direction: column; + gap: 0.18rem; + font-size: clamp(1.45rem, 2.1vw, 2rem); + font-weight: 760; + letter-spacing: 0.01em; + line-height: 1.05; + margin: 0; } .banner-moto { - color: #1eff1c; - text-shadow: - 0 0 10px rgba(30, 255, 28, 0.4), - 0 0 20px rgba(30, 255, 28, 0.2); - letter-spacing: 3px; - font-weight: 800; + color: #ffffff; display: inline-flex; align-items: center; + font-weight: 850; + letter-spacing: 0.18em; + text-shadow: 0 0 18px rgba(30, 255, 28, 0.18); user-select: none; -webkit-user-select: none; } .banner-subtitle { - color: #1eff1c; - text-shadow: - 0 0 10px rgba(30, 255, 28, 0.4), - 0 0 20px rgba(30, 255, 28, 0.2); - font-weight: 500; + color: rgba(247, 248, 244, 0.82); + font-size: 0.76em; + font-weight: 560; + letter-spacing: 0.03em; } -.banner-variant { - color: #18cc17; - font-size: 0.95rem; - margin: 0.85rem 0 0 0; - font-weight: 500; - letter-spacing: 0.5px; - font-style: italic; - transform: translateZ(0); - backface-visibility: hidden; +.banner-positioning { + align-self: center; + min-width: 0; + text-align: center; } .banner-mode-subtitle { color: #1eff1c; - font-size: 0.9rem; - line-height: 1.25; - min-height: 1.125rem; - margin: 0.35rem 0 0 0; - font-weight: 700; - letter-spacing: 0.08em; + display: inline-flex; + font-size: 0.73rem; + line-height: 1.2; + margin: 0.18rem 0 0 0; + min-height: 0.95rem; + font-weight: 780; + letter-spacing: 0.12em; + padding: 0.28rem 0.52rem; + border: 1px solid rgba(30, 255, 28, 0.25); + border-radius: 999px; + background: rgba(30, 255, 28, 0.055); text-transform: uppercase; - text-shadow: - 0 0 10px rgba(30, 255, 28, 0.35), - 0 0 18px rgba(30, 255, 28, 0.18); + text-shadow: 0 0 12px rgba(30, 255, 28, 0.22); } .banner-mode-subtitle--hidden { @@ -229,39 +294,58 @@ code { } .banner-company { - color: #ffffff; - font-size: 0.85rem; - margin: 0.1rem 0 0; - letter-spacing: 1px; - font-weight: 400; + color: rgba(212, 175, 55, 0.92); + font-size: 0.78rem; + font-weight: 640; + letter-spacing: 0.12em; + line-height: 1.2; + margin: 0.38rem 0 0; + text-transform: uppercase; } @media (max-width: 768px) { .app-banner { - padding: 1rem 1.5rem; + padding: 0.5rem 1rem; } - + + .banner-content { + align-items: center; + gap: 0.42rem; + } + + .banner-brand { + gap: 0.7rem; + width: auto; + } + + .banner-logo { + width: 58px; + } + .banner-title { - font-size: 1.35rem; - flex-direction: column; - gap: 0.25rem; + font-size: 1.25rem; + gap: 0.12rem; } - + .banner-moto { - letter-spacing: 2px; + letter-spacing: 0.1em; } - + + .banner-positioning { + max-width: none; + text-align: center; + } + .banner-variant { - font-size: 0.85rem; - margin-top: 0.65rem; + font-size: clamp(0.68rem, 2.7vw, 0.8rem); } .banner-mode-subtitle { - font-size: 0.78rem; + font-size: 0.68rem; } - + .banner-company { - font-size: 0.8rem; + font-size: 0.62rem; } } @@ -868,7 +952,7 @@ button.header-status-chip:focus-visible { background: var(--surface-1); padding: 0.75rem 1rem; padding-left: calc(1rem + 84px); - padding-right: 150px; + padding-right: var(--workflow-panel-safe-area); border-bottom: 1px solid var(--border-subtle); flex-wrap: nowrap; } @@ -945,7 +1029,9 @@ button.header-status-chip:focus-visible { padding: 2rem 3rem; overflow-y: auto; scrollbar-gutter: stable; - max-width: 100%; + width: calc(100% - var(--workflow-panel-safe-area)); + max-width: calc(100% - var(--workflow-panel-safe-area)); + box-sizing: border-box; } .container { @@ -954,19 +1040,24 @@ button.header-status-chip:focus-visible { padding: 0 1rem; } +.settings-overlay-safe-area { + width: 100%; +} + .workflow-main-interface > * { transition: filter var(--transition-normal), opacity var(--transition-normal); } -.workflow-main-interface--running > :not(.autonomous-header):not(.button-group):not(.activity-section):not(.tier3-dialog-overlay) { +.workflow-main-interface--running > :not(.autonomous-header):not(.button-group):not(.prompt-composer-section):not(.activity-section):not(.tier3-dialog-overlay) { filter: brightness(0.72); opacity: 0.78; } .workflow-main-interface--running > .autonomous-header, .workflow-main-interface--running > .button-group, +.workflow-main-interface--running > .prompt-composer-section, .workflow-main-interface--running > .activity-section, .workflow-main-interface--running > .tier3-dialog-overlay { filter: none; @@ -1223,6 +1314,10 @@ button.danger:hover { } @media (max-width: 768px) { + .app { + --workflow-panel-safe-area: 0px; + } + .grid-2, .grid-3 { grid-template-columns: 1fr; } @@ -1230,6 +1325,12 @@ button.danger:hover { .tabs { flex-wrap: wrap; padding-left: 1rem; + padding-right: 1rem; + } + + .tab-content { + width: 100%; + max-width: 100%; } .tab { diff --git a/frontend/src/services/api.js b/frontend/src/services/api.js index 0872ab0..546e8c4 100644 --- a/frontend/src/services/api.js +++ b/frontend/src/services/api.js @@ -10,6 +10,24 @@ let proofStatusCache = null; let proofStatusCacheTime = 0; let proofStatusInFlight = null; +export const API_ERROR_KINDS = Object.freeze({ + BACKEND_UNAVAILABLE: 'backend_unavailable', + STALE_TOKEN: 'stale_token', + BACKEND_VALIDATION: 'backend_validation', + AMBIGUOUS_TRANSPORT: 'ambiguous_transport', + BACKEND_RESPONSE: 'backend_response', +}); + +export class MotoApiError extends Error { + constructor(message, { kind, status = null, details = null, cause = null } = {}) { + super(message, cause ? { cause } : undefined); + this.name = 'MotoApiError'; + this.kind = kind || API_ERROR_KINDS.BACKEND_RESPONSE; + this.status = status; + this.details = details; + } +} + function isProofStatusStartingUp(status) { const version = (status?.lean4_version || status?.lean_version || '').trim(); return Boolean(status?.lean4_enabled && (!version || !status?.workspace_ready)); @@ -115,10 +133,44 @@ async function extractErrorMessage(response, fallbackMessage) { */ async function throwFromResponse(response, fallbackMessage) { const message = await extractErrorMessage(response, fallbackMessage); - const err = new Error(message); - err.status = response.status; - err.statusText = response.statusText; - throw err; + let kind = API_ERROR_KINDS.BACKEND_RESPONSE; + if (response.status === 401) { + kind = API_ERROR_KINDS.STALE_TOKEN; + } else if (response.status === 400 || response.status === 409 || response.status === 422) { + kind = API_ERROR_KINDS.BACKEND_VALIDATION; + } + throw new MotoApiError(message, { + kind, + status: response.status, + details: message, + }); +} + +export async function requestJson(url, options = {}, fallbackMessage = 'Backend request failed') { + const method = String(options.method || 'GET').toUpperCase(); + try { + const response = await fetch(url, options); + if (!response.ok) { + await throwFromResponse(response, fallbackMessage); + } + return await response.json(); + } catch (error) { + if (error instanceof MotoApiError) { + throw error; + } + const isMutation = !['GET', 'HEAD', 'OPTIONS'].includes(method); + throw new MotoApiError( + isMutation + ? `${fallbackMessage}: the request outcome is unknown because the backend response was not received` + : `${fallbackMessage}: backend unavailable`, + { + kind: isMutation + ? API_ERROR_KINDS.AMBIGUOUS_TRANSPORT + : API_ERROR_KINDS.BACKEND_UNAVAILABLE, + cause: error, + }, + ); + } } // Aggregator API @@ -174,34 +226,23 @@ export const api = { // Start aggregator async startAggregator(config) { - const response = await fetch(`${API_BASE}/aggregator/start`, { + return requestJson(`${API_BASE}/aggregator/start`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(config), - }); - if (!response.ok) { - const errorData = await response.json(); - const error = new Error('Failed to start aggregator'); - error.details = errorData.detail; - throw error; - } - return response.json(); + }, 'Failed to start aggregator'); }, // Stop aggregator async stopAggregator() { - const response = await fetch(`${API_BASE}/aggregator/stop`, { + return requestJson(`${API_BASE}/aggregator/stop`, { method: 'POST', - }); - if (!response.ok) throw new Error('Failed to stop aggregator'); - return response.json(); + }, 'Failed to stop aggregator'); }, // Get status async getStatus() { - const response = await fetch(`${API_BASE}/aggregator/status`); - if (!response.ok) throw new Error('Failed to get status'); - return response.json(); + return requestJson(`${API_BASE}/aggregator/status`, {}, 'Failed to get aggregator status'); }, async getAggregatorPrompt() { @@ -246,7 +287,19 @@ export const api = { method: 'POST', body: formData, }); - if (!response.ok) throw new Error('Failed to upload file'); + if (!response.ok) { + await throwFromResponse(response, 'Failed to upload file'); + } + return response.json(); + }, + + async deleteUploadedFile(filename) { + const response = await fetch(`${API_BASE}/aggregator/upload-file/${encodeURIComponent(filename)}`, { + method: 'DELETE', + }); + if (!response.ok) { + await throwFromResponse(response, 'Failed to remove uploaded file'); + } return response.json(); }, @@ -307,18 +360,12 @@ export const compilerAPI = { // Start compiler async start(config) { - const response = await fetch(`${API_BASE}/compiler/start`, { + const data = await requestJson(`${API_BASE}/compiler/start`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(config), - }); - if (!response.ok) { - const errorData = await response.json(); - const error = new Error('Failed to start compiler'); - error.details = errorData.detail; - throw error; - } - return { data: await response.json() }; + }, 'Failed to start compiler'); + return { data }; }, // Test models compatibility @@ -334,18 +381,16 @@ export const compilerAPI = { // Stop compiler async stop() { - const response = await fetch(`${API_BASE}/compiler/stop`, { + const data = await requestJson(`${API_BASE}/compiler/stop`, { method: 'POST', - }); - if (!response.ok) throw new Error('Failed to stop compiler'); - return { data: await response.json() }; + }, 'Failed to stop compiler'); + return { data }; }, // Get status async getStatus() { - const response = await fetch(`${API_BASE}/compiler/status`); - if (!response.ok) throw new Error('Failed to get status'); - return { data: await response.json() }; + const data = await requestJson(`${API_BASE}/compiler/status`, {}, 'Failed to get compiler status'); + return { data }; }, async getPrompt() { @@ -453,24 +498,18 @@ export const compilerAPI = { export const autonomousAPI = { // Start autonomous research async start(config) { - const response = await fetch(`${API_BASE}/auto-research/start`, { + return requestJson(`${API_BASE}/auto-research/start`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(config), - }); - if (!response.ok) { - await throwFromResponse(response, 'Failed to start autonomous research'); - } - return response.json(); + }, 'Failed to start autonomous research'); }, // Stop autonomous research async stop() { - const response = await fetch(`${API_BASE}/auto-research/stop`, { + return requestJson(`${API_BASE}/auto-research/stop`, { method: 'POST', - }); - if (!response.ok) throw new Error('Failed to stop autonomous research'); - return response.json(); + }, 'Failed to stop autonomous research'); }, // Clear all autonomous research data @@ -490,9 +529,7 @@ export const autonomousAPI = { // Get status async getStatus() { - const response = await fetch(`${API_BASE}/auto-research/status`); - if (!response.ok) throw new Error('Failed to get autonomous status'); - return response.json(); + return requestJson(`${API_BASE}/auto-research/status`, {}, 'Failed to get autonomous status'); }, // Get all brainstorms @@ -684,8 +721,11 @@ export const autonomousAPI = { return response.text(); }, - async getProofLibrary(novelOnly = true, scope = 'autonomous') { - const response = await fetch(`${API_BASE}${withProofScope(`/proofs/library?novel_only=${novelOnly}`, scope)}`); + async getProofLibrary(category = 'novel', scope = 'autonomous') { + const normalizedCategory = typeof category === 'boolean' + ? (category ? 'novel' : 'all') + : (category || 'novel'); + const response = await fetch(`${API_BASE}${withProofScope(`/proofs/library?category=${encodeURIComponent(normalizedCategory)}`, scope)}`); if (!response.ok) throw new Error('Failed to get proof library'); return response.json(); }, @@ -1201,24 +1241,15 @@ export const boostAPI = { // LeanOJ Proof Solver API export const leanojAPI = { async start(config) { - const response = await fetch(`${API_BASE}/leanoj/start`, { + return requestJson(`${API_BASE}/leanoj/start`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(config), - }); - if (!response.ok) { - const errorData = await response.json().catch(() => ({})); - const error = new Error('Failed to start Proof Solver'); - error.details = errorData.detail; - throw error; - } - return response.json(); + }, 'Failed to start Proof Solver'); }, async stop() { - const response = await fetch(`${API_BASE}/leanoj/stop`, { method: 'POST' }); - if (!response.ok) throw new Error('Failed to stop Proof Solver'); - return response.json(); + return requestJson(`${API_BASE}/leanoj/stop`, { method: 'POST' }, 'Failed to stop Proof Solver'); }, async clear() { @@ -1231,9 +1262,7 @@ export const leanojAPI = { }, async getStatus() { - const response = await fetch(`${API_BASE}/leanoj/status`); - if (!response.ok) throw new Error('Failed to get Proof Solver status'); - return response.json(); + return requestJson(`${API_BASE}/leanoj/status`, {}, 'Failed to get Proof Solver status'); }, async getMasterProof() { @@ -1300,6 +1329,28 @@ export const workflowAPI = { return response.json(); }, + async getSolutionPath() { + const response = await fetch(`${API_BASE}/workflow/solution-path`); + if (!response.ok) throw new Error('Failed to get solution path'); + return response.json(); + }, + + async resumeSolutionPathProposal({ runId, proposalId, lifecycleGeneration }) { + const response = await fetch(`${API_BASE}/workflow/solution-path/resume`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + run_id: runId, + proposal_id: proposalId, + lifecycle_generation: lifecycleGeneration, + }), + }); + if (!response.ok) { + await throwFromResponse(response, 'Failed to retry solution-path proposal'); + } + return response.json(); + }, + // Get cumulative token usage stats and elapsed time async getTokenStats() { const response = await fetch(`${API_BASE}/token-stats`); @@ -1683,6 +1734,12 @@ export const proofSearchAPI = { return response.json(); }, + async getAssistantLatestPack() { + const response = await fetch(`${API_BASE}/proof-search/assistant/latest-pack`); + if (!response.ok) await throwFromResponse(response, 'Failed to load Assistant proof pack'); + return response.json(); + }, + async search(request) { const response = await fetch(`${API_BASE}/proof-search/search`, { method: 'POST', @@ -1693,8 +1750,10 @@ export const proofSearchAPI = { return response.json(); }, - async getProof(source, proofId, { sessionId = null } = {}) { + async getProof(source, proofId, { searchId = null, runId = null, sessionId = null } = {}) { const params = new URLSearchParams(); + if (searchId) params.append('search_id', searchId); + if (runId) params.append('run_id', runId); if (sessionId) params.append('session_id', sessionId); const response = await fetch( `${API_BASE}/proof-search/proofs/${encodeURIComponent(source)}/${encodeURIComponent(proofId)}${params.toString() ? '?' + params.toString() : ''}` @@ -1703,6 +1762,18 @@ export const proofSearchAPI = { return response.json(); }, + async getAssistantSupportLineage(targetHash, supportSearchId, { offset = 0, limit = 50 } = {}) { + const params = new URLSearchParams({ + offset: String(offset), + limit: String(limit), + }); + const response = await fetch( + `${API_BASE}/proof-search/assistant/targets/${encodeURIComponent(targetHash)}/supports/${encodeURIComponent(supportSearchId)}/lineage?${params.toString()}` + ); + if (!response.ok) await throwFromResponse(response, 'Failed to load Assistant support lineage'); + return response.json(); + }, + async reindex() { const response = await fetch(`${API_BASE}/proof-search/reindex`, { method: 'POST', diff --git a/frontend/src/services/api.lifecycle.test.js b/frontend/src/services/api.lifecycle.test.js new file mode 100644 index 0000000..1231b48 --- /dev/null +++ b/frontend/src/services/api.lifecycle.test.js @@ -0,0 +1,47 @@ +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { + API_ERROR_KINDS, + MotoApiError, + autonomousAPI, + requestJson, +} from './api'; + +describe('lifecycle API errors', () => { + afterEach(() => { + vi.restoreAllMocks(); + }); + + test('classifies unreachable reads as backend unavailable', async () => { + vi.stubGlobal('fetch', vi.fn().mockRejectedValue(new TypeError('fetch failed'))); + + await expect(requestJson('/api/health')).rejects.toMatchObject({ + kind: API_ERROR_KINDS.BACKEND_UNAVAILABLE, + }); + }); + + test('classifies mutation transport failures as ambiguous', async () => { + vi.stubGlobal('fetch', vi.fn().mockRejectedValue(new TypeError('connection reset'))); + + await expect(autonomousAPI.stop()).rejects.toMatchObject({ + kind: API_ERROR_KINDS.AMBIGUOUS_TRANSPORT, + }); + }); + + test.each([ + [401, API_ERROR_KINDS.STALE_TOKEN], + [422, API_ERROR_KINDS.BACKEND_VALIDATION], + ])('classifies HTTP %s responses', async (status, kind) => { + vi.stubGlobal('fetch', vi.fn().mockResolvedValue(new Response( + JSON.stringify({ detail: 'request rejected' }), + { status, headers: { 'Content-Type': 'application/json' } }, + ))); + + try { + await autonomousAPI.start({}); + throw new Error('expected request to fail'); + } catch (error) { + expect(error).toBeInstanceOf(MotoApiError); + expect(error).toMatchObject({ kind, status }); + } + }); +}); diff --git a/frontend/src/utils/activityPersistence.js b/frontend/src/utils/activityPersistence.js new file mode 100644 index 0000000..3d8442c --- /dev/null +++ b/frontend/src/utils/activityPersistence.js @@ -0,0 +1,51 @@ +const REDACTED = '[redacted]'; + +const SENSITIVE_FIELD_PATTERN = /^(?:access[_-]?token|api[_-]?key|app[_-]?id|authorization|bearer|client[_-]?secret|code[_-]?verifier|credential|id[_-]?token|password|refresh[_-]?token|secret|session[_-]?token|token)$/i; +const NAMED_SECRET_PATTERN = /((?:access[_-]?token|api[_-]?key|app[_-]?id|authorization|client[_-]?secret|code[_-]?verifier|id[_-]?token|password|refresh[_-]?token|secret|session[_-]?token|token)\s*["']?\s*[:=]\s*["']?)([^"',\s&#]+)/gi; +const BEARER_PATTERN = /\b(Bearer\s+)[A-Za-z0-9._~+/-]+=*/gi; +const OPENROUTER_KEY_PATTERN = /\bsk-or-v1-[A-Za-z0-9_-]+\b/gi; +const OAUTH_URL_PARAMETER_PATTERN = /([?&#](?:access_token|authorization|client_secret|code|code_verifier|id_token|refresh_token|token)=)([^&#\s]+)/gi; + +export function redactPersistedActivityText(value) { + return String(value ?? '') + .replace(BEARER_PATTERN, `$1${REDACTED}`) + .replace(OPENROUTER_KEY_PATTERN, REDACTED) + .replace(OAUTH_URL_PARAMETER_PATTERN, `$1${REDACTED}`) + .replace(NAMED_SECRET_PATTERN, `$1${REDACTED}`); +} + +export function sanitizePersistedActivityValue(value, seen = new WeakSet()) { + if (value == null || typeof value === 'number' || typeof value === 'boolean') { + return value; + } + if (typeof value === 'string') { + return redactPersistedActivityText(value); + } + if (typeof value !== 'object') { + return redactPersistedActivityText(value); + } + if (seen.has(value)) { + return '[omitted circular value]'; + } + + seen.add(value); + let sanitized; + if (Array.isArray(value)) { + sanitized = value.map((item) => sanitizePersistedActivityValue(item, seen)); + } else { + sanitized = Object.fromEntries( + Object.entries(value).map(([key, nestedValue]) => [ + key, + SENSITIVE_FIELD_PATTERN.test(key) + ? REDACTED + : sanitizePersistedActivityValue(nestedValue, seen), + ]), + ); + } + seen.delete(value); + return sanitized; +} + +export function isSensitivePersistedActivityField(key) { + return SENSITIVE_FIELD_PATTERN.test(String(key || '')); +} diff --git a/frontend/src/utils/activityPersistence.test.js b/frontend/src/utils/activityPersistence.test.js new file mode 100644 index 0000000..3e9a9cb --- /dev/null +++ b/frontend/src/utils/activityPersistence.test.js @@ -0,0 +1,60 @@ +import { describe, expect, test } from 'vitest'; +import { + isSensitivePersistedActivityField, + redactPersistedActivityText, + sanitizePersistedActivityValue, +} from './activityPersistence'; + +describe('activity persistence sanitization', () => { + test('redacts nested credential fields without mutating useful metadata', () => { + const input = { + provider: 'openrouter', + role_id: 'validator', + session_id: 'session-123', + nested: [{ access_token: 'access-secret', model: 'model-a' }], + apiKey: 'key-secret', + }; + + const output = sanitizePersistedActivityValue(input); + + expect(output).toEqual({ + provider: 'openrouter', + role_id: 'validator', + session_id: 'session-123', + nested: [{ access_token: '[redacted]', model: 'model-a' }], + apiKey: '[redacted]', + }); + expect(input.nested[0].access_token).toBe('access-secret'); + }); + + test('redacts credentials embedded in messages and callback URLs', () => { + const text = [ + 'Authorization: Bearer abc.def.ghi', + 'api_key=plain-secret', + 'sk-or-v1-supersecret', + 'https://callback.test/?code=oauth-code&state=keep&refresh_token=refresh-secret', + ].join(' '); + const redacted = redactPersistedActivityText(text); + + expect(redacted).not.toContain('abc.def.ghi'); + expect(redacted).not.toContain('plain-secret'); + expect(redacted).not.toContain('supersecret'); + expect(redacted).not.toContain('oauth-code'); + expect(redacted).not.toContain('refresh-secret'); + expect(redacted).toContain('state=keep'); + }); + + test('does not classify provenance session IDs as session tokens', () => { + expect(isSensitivePersistedActivityField('session_id')).toBe(false); + expect(isSensitivePersistedActivityField('session_token')).toBe(true); + }); + + test('handles circular values without throwing', () => { + const input = { provider: 'openrouter' }; + input.self = input; + expect(sanitizePersistedActivityValue(input)).toEqual({ + provider: 'openrouter', + self: '[omitted circular value]', + }); + }); +}); diff --git a/frontend/src/utils/activityStyles.js b/frontend/src/utils/activityStyles.js index 089915c..d5505e0 100644 --- a/frontend/src/utils/activityStyles.js +++ b/frontend/src/utils/activityStyles.js @@ -2,9 +2,36 @@ export const CONTEXT_OVERFLOW_STOP_MESSAGE = 'Research stopped. Some required so export const REJECTION_FEEDBACK_NOTICE = 'Rejections are normal and provide feedback to the model. Extended rejection streaks can be expected on difficult problems. Above is 10 submissions your validator thought were not worth your time!'; -export const formatContextOverflowActivityMessage = (data = {}) => ( - data.message || CONTEXT_OVERFLOW_STOP_MESSAGE -); +export const formatContextOverflowActivityMessage = (data = {}) => { + const message = data.message || CONTEXT_OVERFLOW_STOP_MESSAGE; + const configuredModel = data.configured_model || ''; + const configuredProvider = data.configured_provider || ''; + const effectiveModel = data.effective_model || data.model || ''; + const effectiveProvider = data.effective_provider || data.provider || ''; + const effectiveHost = data.effective_host_provider || data.host_provider || ''; + const configuredIdentity = [configuredModel, configuredProvider].filter(Boolean).join(' via '); + const effectiveRoute = [ + [effectiveModel, effectiveProvider].filter(Boolean).join(' via '), + effectiveHost ? `host ${effectiveHost}` : '', + ].filter(Boolean).join(', '); + const hasConfigured = Boolean(configuredIdentity); + const hasEffective = Boolean(effectiveRoute); + const routeChanged = hasConfigured && hasEffective && ( + (configuredModel && effectiveModel && configuredModel !== effectiveModel) + || (configuredProvider && effectiveProvider && configuredProvider !== effectiveProvider) + ); + let identity = ''; + if (routeChanged) { + identity = `Effective route: ${effectiveRoute}. Configured route: ${configuredIdentity}.`; + } else if (hasEffective) { + identity = `Route: ${effectiveRoute}.`; + } else if (hasConfigured) { + identity = `Configured route: ${configuredIdentity}.`; + } + if (!identity) return message; + const separator = /[.!?]$/.test(message.trim()) ? ' ' : '. '; + return `${message}${separator}${identity}`; +}; export const shouldAddRejectionFeedbackNotice = (data = {}, observedConsecutiveRejections = null, shown = {}) => { const total = Number(data.total_rejections ?? data.total_rejection_count ?? data.rejection_count); @@ -31,8 +58,49 @@ export const buildRejectionFeedbackNoticeActivity = (timestamp, data = {}) => ({ }, }); +export const formatSolutionPathEventMessage = (event = '', data = {}) => { + if (data.message) return data.message; + const queued = Number(data.queued_proposals || 0); + switch (event) { + case 'solution_path_activated': + return 'Progressive solution-path tracking is now active.'; + case 'solution_path_proposal_queued': + return `A solution-path update was queued for Main Submitter 1 review${queued ? ` (${queued} queued)` : ''}.`; + case 'solution_path_proposal_reviewing': + return 'Main Submitter 1 is reviewing a proposed solution-path update.'; + case 'solution_path_updated': + return `Main Submitter 1 approved solution path revision ${Number(data.revision || 0)}.`; + case 'solution_path_proposal_rejected': + return 'Main Submitter 1 rejected a proposed solution-path update.'; + case 'solution_path_proposal_retry_queued': + return `A solution-path update remains queued for retry${queued ? ` (${queued} queued)` : ''}.`; + case 'solution_path_proposal_user_repair_required': + return 'A solution-path update needs a provider, model, key, privacy, or context setting repaired before review can continue.'; + case 'solution_path_proposal_resumed': + return 'The repaired solution-path update was explicitly returned to the Main Submitter 1 review queue.'; + default: + return 'Solution path changed.'; + } +}; + export const getActivityIcon = (event = '') => { switch (event) { + case 'solution_path_activated': + return '◇'; + case 'solution_path_proposal_queued': + return '+'; + case 'solution_path_proposal_reviewing': + return '◎'; + case 'solution_path_updated': + return '✓'; + case 'solution_path_proposal_rejected': + return '✗'; + case 'solution_path_proposal_retry_queued': + return '↺'; + case 'solution_path_proposal_user_repair_required': + return '⚠'; + case 'solution_path_proposal_resumed': + return '▶'; case 'assistant_proof_pack_updated': return 'A'; case 'assistant_proof_pack_failed': @@ -176,6 +244,7 @@ export const getActivityIcon = (event = '') => { case 'proof_attempts_exhausted': return '⚠'; case 'context_overflow_error': + case 'proof_context_overflow': return '!'; case 'proof_verified': case 'known_proof_verified': @@ -278,11 +347,33 @@ export const getActivityClass = (event = '', item = {}) => { } if ( - event === 'assistant_proof_pack_updated' + event === 'assistant_proof_pack_updated' || + event === 'solution_path_activated' || + event === 'solution_path_proposal_queued' || + event === 'solution_path_proposal_reviewing' ) { return 'activity-info'; } + if (event === 'solution_path_updated') { + return 'activity-success'; + } + + if (event === 'solution_path_proposal_rejected') { + return 'activity-reject'; + } + + if ( + event === 'solution_path_proposal_retry_queued' + || event === 'solution_path_proposal_user_repair_required' + ) { + return 'activity-warning'; + } + + if (event === 'solution_path_proposal_resumed') { + return 'activity-info'; + } + if ( event.includes('accepted') || event === 'compiler_acceptance' || @@ -323,6 +414,7 @@ export const getActivityClass = (event = '', item = {}) => { event === 'proof_integrity_rejected' || event === 'smt_check_error' || event === 'context_overflow_error' || + event === 'proof_context_overflow' || event === 'leanoj_brainstorm_rejected' || event === 'leanoj_brainstorm_submitter_failed' || event === 'leanoj_brainstorm_prune_rejected' || @@ -422,14 +514,20 @@ export const formatAssistantProofPackMessage = (data = {}) => { const selectorText = assistantSelected ? ` via Assistant${assistantModel ? ` (${assistantModel})` : ''}` : (selectionMode ? ` via ${selectionMode}` : ''); - const candidateCount = Number.isFinite(Number(data.candidate_count)) ? Number(data.candidate_count) : null; - const candidateText = candidateCount !== null ? ` from ${candidateCount} candidates` : ''; + const reviewedByCorpus = data.retrieval_observability?.deduped_distinct?.by_corpus || {}; + const reviewedSynthetic = Number(reviewedByCorpus.syntheticlib4 || 0); + const reviewedLocal = Object.entries(reviewedByCorpus) + .filter(([corpus]) => corpus !== 'syntheticlib4') + .reduce((sum, [, count]) => sum + Number(count || 0), 0); + const countsText = Object.keys(reviewedByCorpus).length + ? `: reviewed ${reviewedLocal} local and ${reviewedSynthetic} SyntheticLib4; used ${local} local and ${synthetic} SyntheticLib4` + : `: used ${local} local and ${synthetic} SyntheticLib4`; if (total === 0 && warningCount === 0) { - return `Assistant memory found no useful proofs${candidateText} for ${target}${phaseText}${selectorText}: ${local} local, ${synthetic} SyntheticLib4`; + return `Assistant memory found no useful proofs for ${target}${phaseText}${selectorText}${countsText}`; } - return `Assistant memory returned ${total}/${max} proofs${candidateText} for ${target}${phaseText}${selectorText}: ${local} local, ${synthetic} SyntheticLib4${warningText}`; + return `Assistant memory returned ${total}/${max} proofs for ${target}${phaseText}${selectorText}${countsText}${warningText}`; }; export const ASSISTANT_PROOF_PACK_EVENTS = new Set([ @@ -522,16 +620,17 @@ export const formatAssistantProofPackEventMessage = (event = '', data = {}) => { const assistantModel = String(data.assistant_model_id || '').trim(); const assistantSelected = Boolean(String(data.assistant_role_id || assistantModel || '').trim()); const modelText = assistantSelected ? ` via Assistant${assistantModel ? ` (${assistantModel})` : ''}` : ''; - const candidateCount = Number.isFinite(Number(data.candidate_count)) ? Number(data.candidate_count) : null; - const shortlistCount = Number.isFinite(Number(data.shortlist_count)) ? Number(data.shortlist_count) : null; - const candidateText = candidateCount !== null - ? ` from ${candidateCount} candidates${shortlistCount !== null ? ` (${shortlistCount} shortlisted)` : ''}` + const observability = data.retrieval_observability || {}; + const reviewedByCorpus = observability.deduped_distinct?.by_corpus || {}; + const usedByCorpus = observability.final_selected?.by_corpus || {}; + const sumLocal = (counts) => Object.entries(counts) + .filter(([corpus]) => corpus !== 'syntheticlib4') + .reduce((sum, [, count]) => sum + Number(count || 0), 0); + const counterText = Object.keys(reviewedByCorpus).length + ? `: reviewed ${sumLocal(reviewedByCorpus)} local and ${Number(reviewedByCorpus.syntheticlib4 || 0)} SyntheticLib4; used ${sumLocal(usedByCorpus)} local and ${Number(usedByCorpus.syntheticlib4 || 0)} SyntheticLib4` : ''; if (event === 'assistant_proof_pack_refresh_started') { - if (candidateCount !== null && shortlistCount !== null && assistantSelected) { - return `Assistant memory refresh started for ${target}${phaseText}: local proof-search ranking shortlisted ${shortlistCount} of ${candidateCount} candidates for Assistant review`; - } - return `Assistant memory refresh started${candidateText} for ${target}${phaseText}${modelText}`; + return `Assistant memory refresh started for ${target}${phaseText}${modelText}`; } if (event === 'assistant_proof_pack_warning') { const warnings = Array.isArray(data.warnings) ? data.warnings.filter(Boolean).join('; ') : ''; @@ -540,7 +639,7 @@ export const formatAssistantProofPackEventMessage = (event = '', data = {}) => { if (event === 'assistant_proof_pack_failed') { const detail = String(data.error_message || data.reason || '').trim(); const failureText = detail ? `: ${detail}` : ''; - return `Assistant memory model call failed for ${target}${phaseText}${modelText}${candidateText}${failureText}`; + return `Assistant memory model call failed for ${target}${phaseText}${modelText}${failureText}${counterText}`; } if (event === 'assistant_proof_pack_stopped') { return `Assistant memory stopped (${data.reason || 'parent stopped'})`; diff --git a/frontend/src/utils/activityStyles.test.js b/frontend/src/utils/activityStyles.test.js index 917529b..52021cd 100644 --- a/frontend/src/utils/activityStyles.test.js +++ b/frontend/src/utils/activityStyles.test.js @@ -5,11 +5,67 @@ import { buildRejectionFeedbackNoticeActivity, formatAssistantProofPackEventMessage, formatAssistantProofPackMessage, + formatContextOverflowActivityMessage, + formatSolutionPathEventMessage, getActivityClass, + getActivityIcon, getAssistantProofPackDuplicateKey, shouldAddRejectionFeedbackNotice, } from './activityStyles'; +test('context overflow activity identifies the effective or configured model', () => { + expect(formatContextOverflowActivityMessage({ + message: 'Research stopped.', + configured_model: 'configured/model', + configured_provider: 'openrouter', + })).toBe('Research stopped. Configured route: configured/model via openrouter.'); + + expect(formatContextOverflowActivityMessage({ + message: 'Research stopped.', + configured_model: 'configured/model', + effective_model: 'fallback/model', + effective_provider: 'lm_studio', + })).toBe( + 'Research stopped. Effective route: fallback/model via lm_studio. ' + + 'Configured route: configured/model.' + ); + + expect(formatContextOverflowActivityMessage({ + message: 'Proof formalization skipped.', + configured_model: 'configured/model', + configured_provider: 'openrouter', + })).toBe('Proof formalization skipped. Configured route: configured/model via openrouter.'); +}); + +test('proof context overflow uses fatal activity styling without implying workflow stop', () => { + expect(getActivityClass('proof_context_overflow')).toBe('activity-reject'); +}); + +test('styles and formats every solution-path lifecycle event', () => { + const expectations = { + solution_path_activated: ['◇', 'activity-info'], + solution_path_proposal_queued: ['+', 'activity-info'], + solution_path_proposal_reviewing: ['◎', 'activity-info'], + solution_path_updated: ['✓', 'activity-success'], + solution_path_proposal_rejected: ['✗', 'activity-reject'], + solution_path_proposal_retry_queued: ['↺', 'activity-warning'], + solution_path_proposal_user_repair_required: ['⚠', 'activity-warning'], + solution_path_proposal_resumed: ['▶', 'activity-info'], + }; + + Object.entries(expectations).forEach(([event, [icon, activityClass]]) => { + expect(getActivityIcon(event)).toBe(icon); + expect(getActivityClass(event)).toBe(activityClass); + expect(formatSolutionPathEventMessage(event, {})).not.toBe('Solution path changed.'); + }); + expect(formatSolutionPathEventMessage('solution_path_proposal_queued', { + queued_proposals: 2, + })).toContain('(2 queued)'); + expect(formatSolutionPathEventMessage('solution_path_updated', { + message: 'Engine supplied message.', + })).toBe('Engine supplied message.'); +}); + test('formats clean empty Assistant proof pack as info instead of warning', () => { const message = formatAssistantProofPackMessage({ result_count: 0, @@ -25,7 +81,7 @@ test('formats clean empty Assistant proof pack as info instead of warning', () = }); expect(message).toBe( - 'Assistant memory found no useful proofs from 64 candidates for brainstorm context during brainstorm via Assistant (openai/gpt-oss-20b): 0 local, 0 SyntheticLib4' + 'Assistant memory found no useful proofs for brainstorm context during brainstorm via Assistant (openai/gpt-oss-20b): used 0 local and 0 SyntheticLib4' ); expect(message).not.toContain('warning'); }); @@ -60,11 +116,46 @@ test('formats Assistant model-output failure as an error activity', () => { }; expect(formatAssistantProofPackEventMessage('assistant_proof_pack_failed', data)).toBe( - 'Assistant memory model call failed for brainstorm context during brainstorm via Assistant (google/gemma-4-26b-a4b) from 64 candidates (20 shortlisted): No JSON found in response' + 'Assistant memory model call failed for brainstorm context during brainstorm via Assistant (google/gemma-4-26b-a4b): No JSON found in response' ); expect(getActivityClass('assistant_proof_pack_failed')).toBe('activity-reject'); }); +test('formats federated Assistant lane counts without exposing proof content', () => { + const message = formatAssistantProofPackEventMessage('assistant_proof_pack_updated', { + result_count: 5, + max_result_count: 7, + candidate_count: 32, + shortlist_count: 21, + target_kind: 'paper', + local_result_count: 4, + syntheticlib4_result_count: 1, + retrieval_observability: { + raw_by_lane: { + local: { total: 40 }, + duplicate_neighborhood: { total: 12 }, + syntheticlib4: { total: 8 }, + }, + deduped_distinct: { + total: 48, + by_corpus: { moto: 40, syntheticlib4: 8 }, + }, + fused_cap_64: { total: 32 }, + shortlist_21: { total: 21 }, + final_selected: { + total: 5, + by_corpus: { moto: 4, syntheticlib4: 1 }, + }, + matching_runs_examined: 9, + matching_occurrences_examined: 60, + }, + }); + + expect(message).toContain('reviewed 40 local and 8 SyntheticLib4'); + expect(message).toContain('used 4 local and 1 SyntheticLib4'); + expect(message).not.toContain('64 candidates'); +}); + test('adds rejection feedback notice on first and tenth consecutive rejection only', () => { expect(shouldAddRejectionFeedbackNotice({ total_rejections: 1 })).toBe(true); expect(shouldAddRejectionFeedbackNotice({ total_rejections: 7, consecutive_rejections: 10 })).toBe(true); diff --git a/frontend/src/utils/autonomousProfiles.js b/frontend/src/utils/autonomousProfiles.js index adb0f86..37f49e5 100644 --- a/frontend/src/utils/autonomousProfiles.js +++ b/frontend/src/utils/autonomousProfiles.js @@ -427,8 +427,8 @@ const DEFAULT_AUTONOMOUS_SETTINGS = { submitterConfigs: DEFAULT_OPENROUTER_SUBMITTER_CONFIGS, localConfig: DEFAULT_LOCAL_CONFIG, freeOnly: false, - freeModelLooping: true, - freeModelAutoSelector: true, + freeModelLooping: false, + freeModelAutoSelector: false, allowMathematicalProofs: true, allowResearchPapers: true, tier3Enabled: false, diff --git a/frontend/src/utils/leanojProfiles.js b/frontend/src/utils/leanojProfiles.js index 2637087..ae1a6ca 100644 --- a/frontend/src/utils/leanojProfiles.js +++ b/frontend/src/utils/leanojProfiles.js @@ -145,8 +145,8 @@ const DEFAULT_SETTINGS = { maxRecursiveBrainstormAccepts: 10, finalAttemptsPerCycle: 30, freeOnly: false, - freeModelLooping: true, - freeModelAutoSelector: true, + freeModelLooping: false, + freeModelAutoSelector: false, creativityEmphasisBoostEnabled: false, modelProviders: {}, selectedProfile: LEANOJ_RECOMMENDED_PROFILE_KEY, diff --git a/frontend/src/utils/proofPresentation.js b/frontend/src/utils/proofPresentation.js new file mode 100644 index 0000000..cb60225 --- /dev/null +++ b/frontend/src/utils/proofPresentation.js @@ -0,0 +1,124 @@ +export const STRICT_NOVELTY_CATEGORIES = Object.freeze([ + 'major_mathematical_discovery', + 'mathematical_discovery', + 'novel_variant', + 'novel_formulation', + 'duplicate_novel', + 'not_novel', +]); + +const NOVELTY_PRESENTATION = Object.freeze({ + major_mathematical_discovery: { + label: 'Major Mathematical Discovery', + shortLabel: 'Major Discovery', + badgeClass: 'proof-badge--platinum', + cardClass: 'proof-card--platinum', + tileClass: 'assistant-proof-tile--platinum', + group: 'novel', + }, + mathematical_discovery: { + label: 'Minor Mathematical Discovery', + shortLabel: 'Discovery', + badgeClass: 'proof-badge--gold', + cardClass: 'proof-card--gold', + tileClass: 'assistant-proof-tile--gold', + group: 'novel', + }, + novel_variant: { + label: 'Novel Reformulation', + shortLabel: 'Novel Variant', + badgeClass: 'proof-badge--silver', + cardClass: 'proof-card--silver', + tileClass: 'assistant-proof-tile--silver', + group: 'novel', + }, + novel_formulation: { + label: 'Novel Formalization', + shortLabel: 'Novel Formalization', + badgeClass: 'proof-badge--bronze', + cardClass: 'proof-card--bronze', + tileClass: 'assistant-proof-tile--bronze', + group: 'novel', + }, + duplicate_novel: { + label: 'Duplicate Novel', + shortLabel: 'Duplicate Novel', + badgeClass: 'proof-badge--duplicate-novel', + cardClass: 'proof-card--duplicate-novel', + tileClass: 'assistant-proof-tile--duplicate-novel', + group: 'duplicate_novel', + }, + not_novel: { + label: 'Not Novel', + shortLabel: 'Not Novel', + badgeClass: 'proof-badge--known', + cardClass: 'proof-card--known', + tileClass: 'assistant-proof-tile--known', + group: 'not_novel', + }, + unknown: { + label: 'Unknown (Legacy)', + shortLabel: 'Unknown', + badgeClass: 'proof-badge--known', + cardClass: 'proof-card--known', + tileClass: 'assistant-proof-tile--known', + group: 'unknown', + }, +}); + +export function classifyProofNovelty(proof = {}) { + const category = String(proof.novelty_tier || '').trim().toLowerCase(); + return { + category: STRICT_NOVELTY_CATEGORIES.includes(category) ? category : 'unknown', + ...NOVELTY_PRESENTATION[STRICT_NOVELTY_CATEGORIES.includes(category) ? category : 'unknown'], + }; +} + +export function sanitizeDomId(value, prefix = 'proof') { + const normalized = String(value ?? '') + .normalize('NFKD') + .replace(/[^A-Za-z0-9_-]+/g, '-') + .replace(/^-+|-+$/g, ''); + return `${prefix}-${normalized || 'unknown'}`; +} + +export function getCanonicalProofIdentity(proof = {}, { includeIndex = false, index = 0 } = {}) { + const canonical = String(proof.search_id || '').trim(); + if (canonical) return canonical; + const corpus = String(proof.corpus || proof.scope || 'proof').trim(); + const run = String(proof.run_id || proof.session_id || 'legacy').trim(); + const proofId = String(proof.proof_id || proof.library_id || proof.lean_code_hash || proof.theorem_statement_hash || 'unknown').trim(); + return [corpus, run, proofId, includeIndex ? index : ''].filter((part) => part !== '').join(':'); +} + +export function getLeanOJProofPresentation(proof = {}) { + if (proof.proof_kind === 'final') { + return { + badgeClass: 'proof-badge--gold', + cardClass: 'proof-card--gold', + label: 'Final Verified Submission', + }; + } + return { + badgeClass: 'proof-badge--silver', + cardClass: 'proof-card--silver', + label: 'Verified Proof Fragment', + }; +} + +export function formatProofProvenance(proof = {}) { + const runId = proof.run_id || ''; + const sessionId = proof.session_id || ''; + const source = proof.source_type + ? `${proof.corpus_scope ? `${proof.corpus_scope} · ` : ''}${proof.source_type}${proof.source_id ? `/${proof.source_id}` : ''}` + : ''; + const lanes = Array.isArray(proof.retrieval_lanes) ? proof.retrieval_lanes : []; + const omitted = Number(proof.occurrence_omitted ?? proof.omitted_total ?? 0); + return { + runId, + sessionId, + source, + lanes, + omitted: Number.isFinite(omitted) && omitted > 0 ? omitted : 0, + }; +} diff --git a/frontend/src/utils/proofPresentation.test.js b/frontend/src/utils/proofPresentation.test.js new file mode 100644 index 0000000..9cd0cf6 --- /dev/null +++ b/frontend/src/utils/proofPresentation.test.js @@ -0,0 +1,46 @@ +import { describe, expect, test } from 'vitest'; +import { + classifyProofNovelty, + formatProofProvenance, + sanitizeDomId, +} from './proofPresentation'; + +describe('proof presentation', () => { + test.each([ + ['major_mathematical_discovery', 'novel'], + ['mathematical_discovery', 'novel'], + ['novel_variant', 'novel'], + ['novel_formulation', 'novel'], + ['duplicate_novel', 'duplicate_novel'], + ['not_novel', 'not_novel'], + ])('classifies strict backend category %s', (noveltyTier, group) => { + expect(classifyProofNovelty({ novelty_tier: noveltyTier }).group).toBe(group); + }); + + test('does not infer novelty from legacy boolean fields', () => { + expect(classifyProofNovelty({ novel: true })).toMatchObject({ + category: 'unknown', + label: 'Unknown (Legacy)', + group: 'unknown', + }); + }); + + test('sanitizes stable DOM ids and formats bounded lineage provenance', () => { + expect(sanitizeDomId('manual:run/1 proof#2', 'details')).toBe('details-manual-run-1-proof-2'); + expect(formatProofProvenance({ + run_id: 'run-1', + session_id: 'session-1', + corpus_scope: 'manual', + source_type: 'paper', + source_id: 'paper-1', + retrieval_lanes: ['exact', 'semantic'], + occurrence_omitted: 4, + })).toEqual({ + runId: 'run-1', + sessionId: 'session-1', + source: 'manual · paper/paper-1', + lanes: ['exact', 'semantic'], + omitted: 4, + }); + }); +}); diff --git a/frontend/src/utils/runtimeConfig.js b/frontend/src/utils/runtimeConfig.js index 17d558d..24de217 100644 --- a/frontend/src/utils/runtimeConfig.js +++ b/frontend/src/utils/runtimeConfig.js @@ -9,9 +9,16 @@ function toScopedKey(key) { if (!storagePrefix || typeof key !== 'string' || key.length === 0) { return key; } + if (key.startsWith(`${storagePrefix}:`)) { + return key; + } return `${storagePrefix}:${key}`; } +export function getNamespacedStorageKey(key) { + return toScopedKey(key); +} + export function installNamespacedLocalStorage() { if (typeof window === 'undefined' || !storagePrefix) { return; diff --git a/frontend/src/utils/solutionPathPresentation.js b/frontend/src/utils/solutionPathPresentation.js new file mode 100644 index 0000000..ab1b97d --- /dev/null +++ b/frontend/src/utils/solutionPathPresentation.js @@ -0,0 +1,50 @@ +const asNumber = (value) => { + const parsed = Number(value); + return Number.isFinite(parsed) ? parsed : 0; +}; + +export const isSameSolutionPathRun = (left, right) => ( + Boolean(left?.run_id) + && Boolean(right?.run_id) + && left.run_id === right.run_id +); + +export const isSolutionPathSnapshotAtLeast = (candidate, current) => { + if (!candidate) return false; + if (!current) return true; + if (!isSameSolutionPathRun(candidate, current)) return true; + + const candidateGeneration = asNumber(candidate.lifecycle_generation); + const currentGeneration = asNumber(current.lifecycle_generation); + if (candidateGeneration !== currentGeneration) { + return candidateGeneration > currentGeneration; + } + return asNumber(candidate.revision) >= asNumber(current.revision); +}; + +export const solutionPathEventMatches = (event = {}, snapshot = {}, expectedMode = '') => { + if (event.run_id && snapshot.run_id && event.run_id !== snapshot.run_id) return false; + if ( + event.lifecycle_generation != null + && snapshot.lifecycle_generation != null + && asNumber(event.lifecycle_generation) !== asNumber(snapshot.lifecycle_generation) + ) return false; + + const eventMode = String(event.workflow_mode || event.mode || '').toLowerCase(); + if (!eventMode || !expectedMode) return true; + return eventMode === expectedMode; +}; + +export const getSolutionPathSettingsMode = (snapshot = {}) => ( + snapshot.mode === 'compiler' ? 'aggregator' : snapshot.mode +); + +export const getSolutionPathEmptyLabel = (snapshot = {}) => { + if (snapshot.ownership === 'resumable') { + return 'Resumable run · no approved route yet'; + } + if (snapshot.queued_proposals > 0 || snapshot.reviewing_proposals > 0) { + return 'No approved route yet · update pending'; + } + return 'No approved route yet'; +}; diff --git a/frontend/src/utils/solutionPathPresentation.test.js b/frontend/src/utils/solutionPathPresentation.test.js new file mode 100644 index 0000000..d186d09 --- /dev/null +++ b/frontend/src/utils/solutionPathPresentation.test.js @@ -0,0 +1,31 @@ +import { + getSolutionPathEmptyLabel, + getSolutionPathSettingsMode, + isSolutionPathSnapshotAtLeast, + solutionPathEventMatches, +} from './solutionPathPresentation'; + +test('fences snapshots by run generation and revision', () => { + const current = { run_id: 'run-1', lifecycle_generation: 3, revision: 4 }; + expect(isSolutionPathSnapshotAtLeast({ ...current, revision: 5 }, current)).toBe(true); + expect(isSolutionPathSnapshotAtLeast({ ...current, revision: 3 }, current)).toBe(false); + expect(isSolutionPathSnapshotAtLeast({ ...current, lifecycle_generation: 2, revision: 99 }, current)).toBe(false); + expect(isSolutionPathSnapshotAtLeast({ ...current, lifecycle_generation: 4, revision: 0 }, current)).toBe(true); +}); + +test('filters workflow events by run generation and mode', () => { + const snapshot = { run_id: 'run-1', lifecycle_generation: 3 }; + expect(solutionPathEventMatches({ + run_id: 'run-1', + lifecycle_generation: 3, + workflow_mode: 'compiler', + }, snapshot, 'compiler')).toBe(true); + expect(solutionPathEventMatches({ run_id: 'run-2' }, snapshot, 'compiler')).toBe(false); + expect(solutionPathEventMatches({ run_id: 'run-1', lifecycle_generation: 2 }, snapshot, 'compiler')).toBe(false); + expect(solutionPathEventMatches({ workflow_mode: 'autonomous' }, snapshot, 'compiler')).toBe(false); +}); + +test('routes compiler repairs to aggregator settings and labels resumable no-plan state', () => { + expect(getSolutionPathSettingsMode({ mode: 'compiler' })).toBe('aggregator'); + expect(getSolutionPathEmptyLabel({ ownership: 'resumable' })).toMatch(/Resumable run/); +}); diff --git a/moto-update-manifest.json b/moto-update-manifest.json index 6ca1b63..8db93ec 100644 --- a/moto-update-manifest.json +++ b/moto-update-manifest.json @@ -1,7 +1,7 @@ { "manifest_version": 1, - "version": "1.1.02", + "version": "1.1.03", "build_commit": "40be14c63aee7027f02d36479416e1341385b6e3", "update_channel": "main", - "api_contract_version": "build5-v54" + "api_contract_version": "build5-v73" } diff --git a/moto_launcher.py b/moto_launcher.py index 023338f..023475e 100644 --- a/moto_launcher.py +++ b/moto_launcher.py @@ -7,6 +7,7 @@ import contextlib from dataclasses import dataclass from datetime import datetime +import hashlib import importlib import json import ntpath @@ -1347,7 +1348,7 @@ def _set_smt_env_flags( ) -> None: env["MOTO_SMT_ENABLED"] = "1" if enabled else "0" env["MOTO_Z3_PATH"] = z3_path - env["MOTO_SMT_TIMEOUT"] = env.get("MOTO_SMT_TIMEOUT", "").strip() or "30" + env["MOTO_SMT_TIMEOUT"] = env.get("MOTO_SMT_TIMEOUT", "").strip() or "300" def install_lean4( @@ -1623,21 +1624,78 @@ def install_z3(runtime: InstanceRuntime, env: dict[str, str]) -> None: print() -def npm_output_reports_vulnerabilities(output: str) -> bool: - normalized = (output or "").lower() - return ( - "vulnerabilit" in normalized - and ( - "npm audit fix" in normalized - or "severity" in normalized - or "to address all issues" in normalized - ) +def run_frontend_npm_audit(npm_cmd: str, frontend_path: str) -> bool: + """Run read-only high/critical npm audit and return whether the user should be warned.""" + audit_result = subprocess.run( + [npm_cmd, "audit", "--audit-level=high"], + cwd=frontend_path, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + check=False, + ) + audit_output = (audit_result.stdout or "").strip() + if audit_output: + print(audit_output) + + if audit_result.returncode == 0: + return False + + cprint( + "WARNING: npm reported high/critical frontend dependency advisories. " + "The launcher will not rewrite dependencies locally; the auto-updater will apply the tested fix " + "with the next MOTO update after maintainers update the lockfile.", + YELLOW, ) + return True + + +def file_sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def frontend_lockfile_install_current(frontend_dir: Path, package_lock_path: Path) -> bool: + marker_path = frontend_dir / "node_modules" / ".moto_package_lock.sha256" + if not marker_path.exists(): + return False + try: + return marker_path.read_text(encoding="utf-8").strip() == file_sha256(package_lock_path) + except OSError: + return False + + +def write_frontend_lockfile_install_marker(frontend_dir: Path, package_lock_path: Path) -> None: + marker_path = frontend_dir / "node_modules" / ".moto_package_lock.sha256" + try: + marker_path.parent.mkdir(parents=True, exist_ok=True) + marker_path.write_text(file_sha256(package_lock_path), encoding="utf-8") + except OSError as exc: + cprint(f"WARNING: Could not write frontend dependency marker: {exc}", YELLOW) + + +def npm_output_is_windows_file_lock_error(output: str) -> bool: + normalized = (output or "").lower() + return "eperm" in normalized and ("unlink" in normalized or "operation not permitted" in normalized) + + +def frontend_install_command(npm_cmd: str, frontend_dir: Path, package_lock_path: Path) -> list[str]: + if not package_lock_path.exists(): + return [npm_cmd, "install"] + + node_modules_path = frontend_dir / "node_modules" + if node_modules_path.exists(): + return [npm_cmd, "install", "--package-lock=false", "--no-save"] + return [npm_cmd, "ci"] def install_frontend_dependencies() -> tuple[str, bool]: cprint("[5/8] Checking Node.js dependencies...", YELLOW) - frontend_path = str(SCRIPT_DIR / "frontend") + frontend_dir = SCRIPT_DIR / "frontend" + frontend_path = str(frontend_dir) if not os.path.isdir(frontend_path): print() cprint("============================================================", RED) @@ -1657,8 +1715,31 @@ def install_frontend_dependencies() -> tuple[str, bool]: cprint("Reinstall Node.js from https://nodejs.org/ and ensure npm is included in PATH.", YELLOW) exit_with_pause(1) + package_lock_path = frontend_dir / "package-lock.json" + node_modules_path = frontend_dir / "node_modules" + install_args = frontend_install_command(npm_cmd, frontend_dir, package_lock_path) + if package_lock_path.exists(): + if node_modules_path.exists() and frontend_lockfile_install_current(frontend_dir, package_lock_path): + cprint("Frontend dependencies match package-lock.json; skipping reinstall.", GREEN) + vulnerability_warning = run_frontend_npm_audit(npm_cmd, frontend_path) + cprint("Node.js dependencies up to date", GREEN) + print() + return npm_cmd, vulnerability_warning + if install_args[1] == "ci": + cprint("Installing frontend dependencies from package-lock.json with `npm ci`...", YELLOW) + else: + cprint( + "Reconciling existing frontend dependencies without modifying package files...", + YELLOW, + ) + else: + cprint( + "WARNING: frontend/package-lock.json is missing; falling back to `npm install` for this launch.", + YELLOW, + ) + npm_result = subprocess.run( - [npm_cmd, "install"], + install_args, cwd=frontend_path, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, @@ -1674,59 +1755,17 @@ def install_frontend_dependencies() -> tuple[str, bool]: cprint("ERROR: Failed to install Node.js dependencies", RED) cprint("============================================================", RED) print() + if package_lock_path.exists() and node_modules_path.exists() and npm_output_is_windows_file_lock_error(npm_output): + cprint("A frontend native module appears to be locked by another process.", YELLOW) + cprint("Close running MOTO/Vite/Node processes and launch again.", YELLOW) cprint("Please check:", YELLOW) cprint("- Internet connection is working", YELLOW) cprint("- package.json exists in frontend directory", YELLOW) exit_with_pause(1) + elif package_lock_path.exists(): + write_frontend_lockfile_install_marker(frontend_dir, package_lock_path) - # ────────────────────────────────────────────────────────────────────── - # SECURITY-CRITICAL: Automatic npm vulnerability remediation. - # PERMANENT LAUNCHER INVARIANT: Do not remove, disable, weaken, or bypass - # this npm audit-fix system. If an agent accidentally removes or breaks it, - # the agent must restore it immediately with no exceptions before continuing - # any other launcher/updater work. - # npm audit fix is the program's self-healing mechanism for dependency - # CVEs. Removing it leaves the runtime exposed to known exploits with - # no user-visible signal that remediation has stopped occurring. - # ────────────────────────────────────────────────────────────────────── - vulnerability_warning = npm_output_reports_vulnerabilities(npm_output) - if vulnerability_warning: - print() - cprint("npm reported vulnerabilities — running `npm audit fix`...", YELLOW) - fix_result = subprocess.run( - [npm_cmd, "audit", "fix"], - cwd=frontend_path, - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT, - text=True, - check=False, - ) - fix_output = (fix_result.stdout or "").strip() - if fix_output: - print(fix_output) - - still_vulnerable = npm_output_reports_vulnerabilities(fix_output) - if fix_result.returncode != 0 or still_vulnerable: - cprint("Standard fix insufficient — running `npm audit fix --force`...", YELLOW) - force_result = subprocess.run( - [npm_cmd, "audit", "fix", "--force"], - cwd=frontend_path, - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT, - text=True, - check=False, - ) - force_output = (force_result.stdout or "").strip() - if force_output: - print(force_output) - if force_result.returncode == 0: - cprint("npm audit fix --force completed.", GREEN) - vulnerability_warning = False - else: - cprint("npm audit fix --force could not fully resolve all vulnerabilities.", YELLOW) - else: - cprint("npm audit fix completed.", GREEN) - vulnerability_warning = False + vulnerability_warning = run_frontend_npm_audit(npm_cmd, frontend_path) cprint("Node.js dependencies up to date", GREEN) print() @@ -1804,6 +1843,68 @@ def is_pid_running(pid: int) -> bool: return True +def wait_for_backend_health( + backend_url: str, + backend_service: LaunchedService, + *, + timeout_seconds: float = 45.0, + poll_interval_seconds: float = 0.25, +) -> dict: + """Wait for the launched backend's readiness endpoint or fail with diagnostics.""" + health_url = f"{backend_url.rstrip('/')}/api/health" + deadline = time.monotonic() + timeout_seconds + last_error = "no response" + while time.monotonic() < deadline: + if not is_pid_running(backend_service.pid): + log_hint = f" Check backend log: {backend_service.log_path}" if backend_service.log_path else "" + raise RuntimeError(f"{backend_service.title} exited before becoming healthy.{log_hint}") + try: + request = Request(health_url, headers={"Accept": "application/json"}) + with urlopen(request, timeout=min(2.0, max(0.1, deadline - time.monotonic()))) as response: + payload = json.loads(response.read().decode("utf-8")) + if response.status == 200 and payload.get("status") == "healthy": + return payload + last_error = f"unexpected health response: {payload!r}" + except (OSError, URLError, TimeoutError, ValueError, json.JSONDecodeError) as exc: + last_error = str(exc) + time.sleep(poll_interval_seconds) + log_hint = f" Backend log: {backend_service.log_path}" if backend_service.log_path else "" + raise RuntimeError( + f"{backend_service.title} did not become healthy within {timeout_seconds:g} seconds " + f"({last_error}).{log_hint}" + ) + + +def wait_for_frontend_ready( + frontend_url: str, + frontend_service: LaunchedService, + *, + timeout_seconds: float = 30.0, + poll_interval_seconds: float = 0.25, +) -> None: + """Wait for Vite to serve the UI, detecting an early launcher child exit.""" + deadline = time.monotonic() + timeout_seconds + last_error = "no response" + while time.monotonic() < deadline: + if not is_pid_running(frontend_service.pid): + log_hint = f" Check frontend log: {frontend_service.log_path}" if frontend_service.log_path else "" + raise RuntimeError(f"{frontend_service.title} exited before becoming ready.{log_hint}") + try: + request = Request(frontend_url, headers={"Accept": "text/html"}) + with urlopen(request, timeout=min(2.0, max(0.1, deadline - time.monotonic()))) as response: + if response.status == 200: + return + last_error = f"HTTP {response.status}" + except (OSError, URLError, TimeoutError) as exc: + last_error = str(exc) + time.sleep(poll_interval_seconds) + log_hint = f" Frontend log: {frontend_service.log_path}" if frontend_service.log_path else "" + raise RuntimeError( + f"{frontend_service.title} did not become ready within {timeout_seconds:g} seconds " + f"({last_error}).{log_hint}" + ) + + def start_services( runtime: InstanceRuntime, env: dict[str, str], @@ -1829,10 +1930,6 @@ def start_services( if not has_desktop_session(): cprint("No DISPLAY/WAYLAND desktop session is active, so you may need to open the frontend URL manually.", YELLOW) print() - cprint("Starting services automatically in 3 seconds...", YELLOW) - time.sleep(3) - print() - backend_args = [ get_python_command(), "-m", @@ -1853,11 +1950,9 @@ def start_services( log_root=runtime.log_root, ) - cprint("Waiting for backend to initialize...", YELLOW) - time.sleep(5) - if backend_service.mode != "window" and not is_pid_running(backend_service.pid): - log_hint = f" Check {backend_service.log_path} for details." if backend_service.log_path else "" - raise RuntimeError(f"{backend_service.title} exited during startup.{log_hint}") + cprint("Waiting for backend health check...", YELLOW) + wait_for_backend_health(backend_url, backend_service) + cprint("Backend is healthy.", GREEN) if runtime.is_default and not runtime.explicit_override: write_runtime_lock(runtime.data_root, backend_service.pid, runtime.instance_id, runtime.backend_port) @@ -1871,11 +1966,9 @@ def start_services( log_root=runtime.log_root, ) - cprint("Waiting for frontend to initialize...", YELLOW) - time.sleep(8) - if frontend_service.mode != "window" and not is_pid_running(frontend_service.pid): - log_hint = f" Check {frontend_service.log_path} for details." if frontend_service.log_path else "" - raise RuntimeError(f"{frontend_service.title} exited during startup.{log_hint}") + cprint("Waiting for frontend readiness...", YELLOW) + wait_for_frontend_ready(frontend_url, frontend_service) + cprint("Frontend is ready.", GREEN) register_active_instance( instance_id=runtime.instance_id, @@ -1885,7 +1978,6 @@ def start_services( frontend_port=runtime.frontend_port, data_root=runtime.data_root, log_root=runtime.log_root, - keyring_namespace=runtime.secret_namespace, storage_prefix=runtime.storage_prefix, ) @@ -1908,7 +2000,6 @@ def start_services( instance_id=runtime.instance_id, data_root=runtime.data_root, log_root=runtime.log_root, - keyring_namespace=runtime.secret_namespace, storage_prefix=runtime.storage_prefix, ) except OSError as exc: @@ -1948,7 +2039,13 @@ def print_success_footer( cprint(f" {frontend_url}", CYAN) print() if vulnerability_warning: - cprint("npm audit fix could not fully resolve all reported vulnerabilities. Review with `npm audit` in `frontend/`.", YELLOW) + cprint( + "npm audit reported high/critical frontend advisories. " + "The launcher did not mutate dependencies; the auto-updater will apply the tested fix " + "with the next MOTO update after maintainers update the lockfile.", + YELLOW, + ) + cprint("For details, run `npm audit --audit-level=high` in `frontend/`.", YELLOW) print() if backend_service.mode == "background" or frontend_service.mode == "background": cprint(f"To stop this instance: stop the launcher-managed backend/frontend processes for {runtime.instance_id}.", YELLOW) diff --git a/moto_updater.py b/moto_updater.py index 0a9fb59..21561d7 100644 --- a/moto_updater.py +++ b/moto_updater.py @@ -32,7 +32,7 @@ "version": "0.0.0-dev", "build_commit": "dev", "update_channel": "main", - "api_contract_version": "build5-v54", + "api_contract_version": "build5-v73", } _DEFAULT_PRESERVED_ROOTS = { @@ -156,7 +156,35 @@ def _read_json(path: Path) -> dict | None: return None -def _write_json(path: Path, payload: dict) -> None: +_FORBIDDEN_METADATA_KEYS = frozenset( + { + "access_token", + "api_key", + "authorization", + "client_secret", + "id_token", + "password", + "refresh_token", + "token", + } +) + + +def _assert_nonsecret_metadata(value: object) -> None: + """Reject credential-bearing fields from launcher/update metadata files.""" + if isinstance(value, dict): + for key, nested_value in value.items(): + if str(key).strip().lower() in _FORBIDDEN_METADATA_KEYS: + raise ValueError(f"Credential field {key!r} is not valid updater metadata") + _assert_nonsecret_metadata(nested_value) + elif isinstance(value, (list, tuple)): + for item in value: + _assert_nonsecret_metadata(item) + + +def _write_nonsecret_json_metadata(path: Path, payload: dict) -> None: + """Write public build or launcher metadata; credential fields are forbidden.""" + _assert_nonsecret_metadata(payload) path.write_text(json.dumps(payload, indent=2), encoding="utf-8") @@ -232,7 +260,7 @@ def _archived_git_head() -> str | None: def _write_installed_manifest(manifest: BuildManifest) -> None: - _write_json( + _write_nonsecret_json_metadata( LOCAL_MANIFEST_PATH, { "manifest_version": manifest.manifest_version, @@ -462,10 +490,30 @@ def _load_launcher_state() -> dict: def _save_launcher_state(payload: dict) -> None: - if not payload.get("instances"): + public_instances = [] + allowed_fields = ( + "instance_id", + "backend_window_pid", + "frontend_window_pid", + "backend_port", + "frontend_port", + "data_root", + "log_root", + "storage_prefix", + ) + for instance in payload.get("instances", []): + if isinstance(instance, dict): + public_instances.append({ + field: instance.get(field) + for field in allowed_fields + }) + if not public_instances: cleanup_path(LAUNCHER_STATE_PATH) return - _write_json(LAUNCHER_STATE_PATH, payload) + _write_nonsecret_json_metadata( + LAUNCHER_STATE_PATH, + {"instances": public_instances}, + ) def cleanup_launcher_state(exclude_instance_id: str | None = None) -> list[dict]: @@ -477,11 +525,7 @@ def cleanup_launcher_state(exclude_instance_id: str | None = None) -> list[dict] backend_pid = _coerce_int(instance.get("backend_window_pid")) frontend_pid = _coerce_int(instance.get("frontend_window_pid")) if _is_pid_running(backend_pid) or _is_pid_running(frontend_pid): - normalized = dict(instance) - keyring_namespace = _record_keyring_namespace(normalized) - normalized.pop("secret_namespace", None) - normalized["keyring_namespace"] = keyring_namespace - active_instances.append(normalized) + active_instances.append(dict(instance)) _save_launcher_state({"instances": active_instances}) if not exclude_instance_id: @@ -502,7 +546,6 @@ def register_active_instance( frontend_port: int, data_root: str, log_root: str, - keyring_namespace: str | None, storage_prefix: str | None, ) -> None: active_instances = cleanup_launcher_state() @@ -549,11 +592,10 @@ def save_last_instance_record( instance_id: str, data_root: str, log_root: str, - keyring_namespace: str | None, storage_prefix: str | None, ) -> None: """Persist the last launched non-default instance so it can be reused on relaunch.""" - _write_json( + _write_nonsecret_json_metadata( LAUNCHER_LAST_INSTANCE_PATH, { "instance_id": instance_id, @@ -760,7 +802,7 @@ def write_update_notice(result: UpdateCheckResult) -> None: "can_auto_apply": result.can_apply_update, "message": build_warning_message(result) if not result.can_apply_update else build_update_prompt(result), } - _write_json(UPDATE_NOTICE_PATH, payload) + _write_nonsecret_json_metadata(UPDATE_NOTICE_PATH, payload) def build_update_prompt(result: UpdateCheckResult) -> str: diff --git a/package-lock.json b/package-lock.json index 986c5e2..a2596a2 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "moto-math-variant", - "version": "1.1.02", + "version": "1.1.03", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "moto-math-variant", - "version": "1.1.02", + "version": "1.1.03", "license": "MIT" } } diff --git a/package.json b/package.json index 8004c13..613b0c5 100644 --- a/package.json +++ b/package.json @@ -1,12 +1,20 @@ { "name": "moto-math-variant", - "version": "1.1.02", - "description": "MOTO S.T.E.M. Mathematics Variant - Autonomous ASI Research System for Novel S.T.E.M. Mathematical Paper Generation", + "version": "1.1.03", + "description": "MOTO Autonomous ASI - multi-agent autonomous research and rigorous solution generation with optional Lean 4 proof verification", "scripts": { "dev:backend": "python -c \"import os, uvicorn; uvicorn.run('backend.api.main:app', host=os.getenv('MOTO_BACKEND_HOST', os.getenv('HOST', '0.0.0.0')), port=int(os.getenv('MOTO_BACKEND_PORT', os.getenv('PORT', '8000'))), reload=True, access_log=False)\"", "dev:frontend": "npm --prefix frontend run dev", "install:frontend": "npm --prefix frontend install", - "build:frontend": "npm --prefix frontend run build" + "build:frontend": "npm --prefix frontend run build", + "test": "npm run test:all", + "test:backend": "python -m pytest", + "test:frontend": "npm --prefix frontend run test", + "test:browser": "npm --prefix frontend run test:browser", + "test:workflows": "python -m pytest tests/workflow_scenarios tests/workflow_recombinatory tests/workflow_real_adapters tests/workflow_cross_field", + "test:context-overflow": "python -m pytest tests/regressions/test_context_overflow_metadata.py tests/regressions/test_context_overflow_emitters.py tests/regressions/test_provider_routing_errors.py tests/regressions/test_proof_context_overflow_metadata.py && npm --prefix frontend run test -- src/utils/activityStyles.test.js src/components/contextOverflowBehavior.test.jsx", + "test:deep": "python tests/run_deep_workflows.py", + "test:all": "npm run test:backend && npm run test:frontend" }, "keywords": [ "ai", diff --git a/requirements.txt b/requirements.txt index 5f1af5b..be0ca83 100644 --- a/requirements.txt +++ b/requirements.txt @@ -5,7 +5,7 @@ python-multipart>=0.0.6 websockets>=12.0 # RAG and embeddings -chromadb>=0.4.22 +chromadb>=0.4.22,<=1.5.9 sentence-transformers>=2.3.1 rank-bm25>=0.2.2 diff --git a/tests/WORKFLOW_TESTING_FUTURE_BUILDS.md b/tests/WORKFLOW_TESTING_FUTURE_BUILDS.md new file mode 100644 index 0000000..44c866f --- /dev/null +++ b/tests/WORKFLOW_TESTING_FUTURE_BUILDS.md @@ -0,0 +1,10 @@ +# Workflow Testing Future Builds + +This historical tracker has been superseded by +`tests/workflow_testing_plans/07_future_builds.md`. + +The wider adapters, deterministic cross-field and deep runs, RAG/event/filesystem +coverage, bounded browser smokes, and backend test organization described here have +landed in bounded form. Remaining blocked seams and later-depth work are maintained in +the Part 7 plan so workflow-test status has one authoritative ledger. + diff --git a/tests/integration/__init__.py b/tests/integration/__init__.py new file mode 100644 index 0000000..7e23001 --- /dev/null +++ b/tests/integration/__init__.py @@ -0,0 +1,2 @@ +"""Route, provider, persistence, and cross-module integration tests.""" + diff --git a/tests/test_allowed_outputs.py b/tests/integration/test_allowed_outputs.py similarity index 66% rename from tests/test_allowed_outputs.py rename to tests/integration/test_allowed_outputs.py index d8fe38a..387486a 100644 --- a/tests/test_allowed_outputs.py +++ b/tests/integration/test_allowed_outputs.py @@ -1,3 +1,4 @@ +import importlib from unittest import IsolatedAsyncioTestCase, mock from fastapi import HTTPException @@ -5,6 +6,7 @@ from backend.api.routes import autonomous as autonomous_route from backend.api.routes import compiler as compiler_route from backend.api.routes import proofs as proofs_route +from backend.autonomous.core.autonomous_coordinator import AutonomousCoordinator from backend.shared.config import system_config from backend.shared.models import ( AutonomousResearchStartRequest, @@ -13,6 +15,140 @@ SubmitterConfig, ) +autonomous_coordinator_module = importlib.import_module( + "backend.autonomous.core.autonomous_coordinator" +) +api_client_manager_module = importlib.import_module("backend.shared.api_client_manager") + + +class AutonomousProofFramingAllowedOutputTests(IsolatedAsyncioTestCase): + async def test_proof_framing_gate_is_not_called_when_proof_output_is_disabled(self) -> None: + coordinator = AutonomousCoordinator() + coordinator._allow_mathematical_proofs = False + coordinator._proof_framing_active = True + coordinator._proof_framing_context = "stale proof emphasis" + + with mock.patch.object( + autonomous_coordinator_module.api_client_manager, + "generate_completion", + new=mock.AsyncMock(), + ) as generate_completion: + await coordinator._run_proof_framing_gate() + + generate_completion.assert_not_awaited() + self.assertFalse(coordinator._proof_framing_active) + self.assertEqual(coordinator._proof_framing_context, "") + + def test_disabled_proof_output_blocks_persisted_framing_injection(self) -> None: + coordinator = AutonomousCoordinator() + coordinator._allow_mathematical_proofs = False + coordinator._proof_framing_active = True + coordinator._proof_framing_context = "stale proof emphasis" + + self.assertEqual(coordinator._append_proof_framing("Research prompt"), "Research prompt") + + def test_lightweight_legacy_coordinator_defaults_to_proof_enabled_boundary(self) -> None: + coordinator = AutonomousCoordinator.__new__(AutonomousCoordinator) + coordinator._proof_framing_active = True + coordinator._proof_framing_context = "Proof emphasis" + + framed = coordinator._append_proof_framing("Research prompt") + + self.assertIn("Research prompt", framed) + self.assertIn("Proof emphasis", framed) + + async def test_disabled_proof_output_blocks_all_proof_context_injection(self) -> None: + coordinator = AutonomousCoordinator() + coordinator._allow_mathematical_proofs = False + coordinator._current_topic_id = "topic-1" + + with ( + mock.patch.object( + autonomous_coordinator_module.proof_database, + "inject_into_prompt", + ) as inject_verified, + mock.patch.object( + autonomous_coordinator_module.proof_database, + "inject_failure_hints_into_prompt", + new=mock.AsyncMock(), + ) as inject_failures, + mock.patch.object( + autonomous_coordinator_module.proof_database, + "count_proofs", + ) as count_proofs, + ): + effective_user = coordinator._get_effective_user_research_prompt() + effective_brainstorm = await coordinator._get_effective_brainstorm_prompt( + "Brainstorm prompt" + ) + + self.assertEqual(effective_user, "") + self.assertEqual(effective_brainstorm, "Brainstorm prompt") + inject_verified.assert_not_called() + inject_failures.assert_not_awaited() + count_proofs.assert_not_called() + + async def test_resume_does_not_restore_persisted_framing_when_proofs_are_disabled(self) -> None: + coordinator = AutonomousCoordinator() + coordinator._allow_mathematical_proofs = False + coordinator._user_research_prompt = "Research prompt" + persisted_state = { + "current_tier": "tier1_aggregation", + "proof_framing_active": True, + "proof_framing_context": "stale proof emphasis", + "proof_framing_reasoning": "Previously enabled", + } + metadata = mock.Mock() + metadata.has_interrupted_workflow.return_value = True + metadata.get_workflow_state = mock.AsyncMock(return_value=persisted_state) + metadata.get_base_user_prompt = mock.AsyncMock(return_value="Research prompt") + metadata.set_proof_framing_state = mock.AsyncMock() + + with ( + mock.patch.object( + coordinator, + "_normalize_resume_state", + new=mock.AsyncMock(return_value=persisted_state), + ), + mock.patch.object( + autonomous_coordinator_module, + "research_metadata", + metadata, + ), + ): + await coordinator._check_resume_state() + + self.assertFalse(coordinator._proof_framing_active) + self.assertEqual(coordinator._proof_framing_context, "") + self.assertEqual(coordinator._append_proof_framing("Research prompt"), "Research prompt") + metadata.set_proof_framing_state.assert_awaited_once() + self.assertFalse(metadata.set_proof_framing_state.await_args.kwargs["active"]) + + async def test_papers_only_run_suppresses_assistant_proof_memory(self) -> None: + manager = autonomous_coordinator_module.api_client_manager + manager.set_assistant_memory_suppressed("autonomous_allowed_outputs", True) + try: + with mock.patch.object( + api_client_manager_module.assistant_proof_search_coordinator, + "submit_target", + ) as submit_target: + messages, target_hash = await manager._maybe_add_assistant_memory_context( + task_id="agg_sub1_001", + role_id="autonomous_topic_selector", + role_config=mock.Mock(), + messages=[{"role": "user", "content": "Research prompt"}], + max_tokens=100, + tools=None, + tool_choice=None, + workflow_mode_override="autonomous", + ) + finally: + manager.set_assistant_memory_suppressed("autonomous_allowed_outputs", False) + + self.assertEqual(messages, [{"role": "user", "content": "Research prompt"}]) + self.assertEqual(target_hash, "") + submit_target.assert_not_called() + class AllowedOutputRouteTests(IsolatedAsyncioTestCase): def setUp(self) -> None: @@ -22,6 +158,8 @@ def setUp(self) -> None: def tearDown(self) -> None: system_config.generic_mode = self._generic_mode system_config.lean4_enabled = self._lean_enabled + compiler_route.compiler_coordinator.is_running = False + compiler_route._release_compiler_workflow_lease() def _compiler_request(self, **overrides) -> CompilerStartRequest: data = { @@ -138,10 +276,16 @@ async def test_compiler_generic_paper_run_downgrades_proof_output(self) -> None: system_config.generic_mode = True system_config.lean4_enabled = False initialize = mock.AsyncMock() + async def mark_started(): + compiler_route.compiler_coordinator.is_running = True with mock.patch.object(compiler_route, "_get_start_conflict", return_value=None): with mock.patch.object(compiler_route.compiler_coordinator, "initialize", initialize): - with mock.patch.object(compiler_route.compiler_coordinator, "start", mock.AsyncMock()): + with mock.patch.object( + compiler_route.compiler_coordinator, + "start", + mock.AsyncMock(side_effect=mark_started), + ): response = await compiler_route.start_compiler(self._compiler_request()) self.assertEqual(response["status"], "started") @@ -236,6 +380,30 @@ async def test_autonomous_generic_paper_run_downgrades_proof_output(self) -> Non self.assertTrue(response["success"]) self.assertFalse(initialize.await_args.kwargs["allow_mathematical_proofs"]) + async def test_autonomous_desktop_papers_only_run_preserves_disabled_proof_output(self) -> None: + system_config.generic_mode = False + system_config.lean4_enabled = True + initialize = mock.AsyncMock() + + with mock.patch.object(autonomous_route, "_get_start_conflict", return_value=None): + with mock.patch.object(autonomous_route, "require_embedding_provider_ready", mock.AsyncMock()): + with mock.patch.object(autonomous_route.autonomous_coordinator, "initialize", initialize): + with mock.patch.object( + autonomous_route.autonomous_coordinator, + "start_in_background", + return_value=True, + ): + response = await autonomous_route.start_autonomous_research( + self._autonomous_request( + allow_mathematical_proofs=False, + allow_research_papers=True, + ) + ) + + self.assertTrue(response["success"]) + self.assertFalse(initialize.await_args.kwargs["allow_mathematical_proofs"]) + self.assertTrue(initialize.await_args.kwargs["allow_research_papers"]) + class ProofStatusReadinessTests(IsolatedAsyncioTestCase): async def test_status_marks_manual_check_unready_when_workspace_is_not_ready(self) -> None: diff --git a/tests/test_assistant_role_schemas.py b/tests/integration/test_assistant_role_schemas.py similarity index 95% rename from tests/test_assistant_role_schemas.py rename to tests/integration/test_assistant_role_schemas.py index dd5acd5..556f909 100644 --- a/tests/test_assistant_role_schemas.py +++ b/tests/integration/test_assistant_role_schemas.py @@ -181,6 +181,12 @@ def test_compiler_start_request_allows_omitted_deprecated_critique_role(self) -> class AssistantRoleDefaultingTests(unittest.IsolatedAsyncioTestCase): + def tearDown(self) -> None: + aggregator_route.coordinator.is_running = False + compiler_route.compiler_coordinator.is_running = False + aggregator_route._release_aggregator_workflow_lease() + compiler_route._release_compiler_workflow_lease() + async def test_aggregator_route_defaults_omitted_assistant_to_validator_config(self) -> None: request = AggregatorStartRequest( user_prompt="Aggregate.", @@ -201,7 +207,13 @@ async def test_aggregator_route_defaults_omitted_assistant_to_validator_config(s mock.patch.object(aggregator_route, "require_embedding_provider_ready", new=mock.AsyncMock()), mock.patch.object(aggregator_route.api_client_manager, "configure_role") as configure_role, mock.patch.object(aggregator_route.coordinator, "initialize", new=mock.AsyncMock()), - mock.patch.object(aggregator_route.coordinator, "start", new=mock.AsyncMock()), + mock.patch.object( + aggregator_route.coordinator, + "start", + new=mock.AsyncMock( + side_effect=lambda: setattr(aggregator_route.coordinator, "is_running", True) + ), + ), mock.patch.object(aggregator_route.token_tracker, "reset"), mock.patch.object(aggregator_route.token_tracker, "start_timer"), ): @@ -250,7 +262,15 @@ async def test_compiler_route_defaults_omitted_assistant_to_validator_config(sel mock.patch.object(compiler_route, "require_embedding_provider_ready", new=mock.AsyncMock()), mock.patch.object(compiler_route.api_client_manager, "configure_role") as configure_role, mock.patch.object(compiler_route.compiler_coordinator, "initialize", new=mock.AsyncMock()), - mock.patch.object(compiler_route.compiler_coordinator, "start", new=mock.AsyncMock()), + mock.patch.object( + compiler_route.compiler_coordinator, + "start", + new=mock.AsyncMock( + side_effect=lambda: setattr( + compiler_route.compiler_coordinator, "is_running", True + ) + ), + ), mock.patch.object(compiler_route.token_tracker, "reset"), mock.patch.object(compiler_route.token_tracker, "start_timer"), ): @@ -295,7 +315,15 @@ async def test_compiler_route_mirrors_deprecated_critique_fields_from_rigor_conf mock.patch.object(compiler_route, "require_embedding_provider_ready", new=mock.AsyncMock()), mock.patch.object(compiler_route.api_client_manager, "configure_role"), mock.patch.object(compiler_route.compiler_coordinator, "initialize", new=mock.AsyncMock()) as initialize, - mock.patch.object(compiler_route.compiler_coordinator, "start", new=mock.AsyncMock()), + mock.patch.object( + compiler_route.compiler_coordinator, + "start", + new=mock.AsyncMock( + side_effect=lambda: setattr( + compiler_route.compiler_coordinator, "is_running", True + ) + ), + ), mock.patch.object(compiler_route.token_tracker, "reset"), mock.patch.object(compiler_route.token_tracker, "start_timer"), ): diff --git a/tests/test_cloud_access_codex.py b/tests/integration/test_cloud_access_codex.py similarity index 98% rename from tests/test_cloud_access_codex.py rename to tests/integration/test_cloud_access_codex.py index e284a78..c221d98 100644 --- a/tests/test_cloud_access_codex.py +++ b/tests/integration/test_cloud_access_codex.py @@ -390,6 +390,20 @@ def json(self): self.assertEqual(fake_http.payload["reasoning"]["effort"], "high") self.assertEqual(response["model"], "gpt-5.3-codex-spark-high") + def test_resolve_gpt_56_named_variants(self) -> None: + self.assertEqual( + OpenAICodexClient._resolve_model_request("gpt-5.6-sol", None), + ("gpt-5.6-sol", None), + ) + self.assertEqual( + OpenAICodexClient._resolve_model_request("gpt-5.6-luna", None), + ("gpt-5.6-sol", "high"), + ) + self.assertEqual( + OpenAICodexClient._resolve_model_request("gpt-5.6-terra", None), + ("gpt-5.6-sol", "medium"), + ) + async def test_list_models_retries_with_newer_stored_token_after_revocation(self) -> None: client = OpenAICodexClient() old_tokens = {"access_token": "old-access", "refresh_token": "refresh"} diff --git a/tests/test_leanoj_coordinator.py b/tests/integration/test_leanoj_coordinator.py similarity index 96% rename from tests/test_leanoj_coordinator.py rename to tests/integration/test_leanoj_coordinator.py index f547056..96a0397 100644 --- a/tests/test_leanoj_coordinator.py +++ b/tests/integration/test_leanoj_coordinator.py @@ -70,6 +70,31 @@ async def fake_review(*_args, **_kwargs): coordinator._review_final_solution_completion = fake_review # type: ignore[method-assign] return coordinator + async def test_solution_path_restores_from_cumulative_acceptance_events(self) -> None: + coordinator = LeanOJCoordinator() + coordinator.get_state().session_id = "leanoj-solution-path-test" + coordinator.get_state().accepted_brainstorm_count = 1 + coordinator.get_state().brainstorm_acceptance_events = 7 + observed: list[int] = [] + + class FakeManager: + async def set_acceptance_count(self, count: int) -> None: + observed.append(count) + + async def fake_acquire(*_args, **_kwargs): + return FakeManager() + + from backend.shared.solution_path import solution_path_registry + + original_acquire = solution_path_registry.acquire + solution_path_registry.acquire = fake_acquire # type: ignore[method-assign] + try: + await coordinator._initialize_solution_path_manager(_request()) + finally: + solution_path_registry.acquire = original_acquire # type: ignore[method-assign] + + self.assertEqual(observed, [7]) + def test_final_solver_prompt_has_no_phase_transition_contract(self) -> None: prompt = build_final_solver_prompt( "Prove one equals one.", @@ -1205,6 +1230,75 @@ async def test_skip_brainstorm_sets_state_flag(self) -> None: finally: system_config.data_dir = old_data_dir + async def test_skip_generation_fences_inflight_brainstorm_submitter(self) -> None: + coordinator = await self._initialized_coordinator() + request = _request() + queue = leanoj_module._LeanOJBrainstormSubmissionQueue(submitter_count=1) + model_started = asyncio.Event() + release_model = asyncio.Event() + + async def fake_context(*_args, **_kwargs): + return {} + + async def blocked_call(*_args, **_kwargs): + model_started.set() + await release_model.wait() + return {"submission": "stale brainstorm result", "reasoning": "stale"} + + coordinator._build_context_blocks = fake_context # type: ignore[method-assign] + coordinator._call_json = blocked_call # type: ignore[method-assign] + task = asyncio.create_task( + coordinator._brainstorm_submitter_loop( # type: ignore[attr-defined] + request, + 1, + request.brainstorm_submitters[0], + queue, + ) + ) + await model_started.wait() + + await coordinator.skip_brainstorm() + release_model.set() + await asyncio.wait_for(task, timeout=1) + + self.assertEqual(queue.qsize(), 0) + self.assertTrue(coordinator.get_state().skip_brainstorm_requested) + + async def test_force_generation_fences_inflight_prune_review(self) -> None: + coordinator = await self._initialized_coordinator() + coordinator._accepted_ideas = ["keep this idea"] # type: ignore[attr-defined] + coordinator._accepted_idea_records = [ # type: ignore[attr-defined] + {"content": "keep this idea", "phase": "recursive_brainstorm", "submitter_index": 1} + ] + review_started = asyncio.Event() + release_review = asyncio.Event() + + async def fake_context(*_args, **_kwargs): + return {} + + async def blocked_review(*_args, **_kwargs): + review_started.set() + await release_review.wait() + return {"action": "delete", "idea_index": 1, "reasoning": "stale"} + + coordinator._build_context_blocks = fake_context # type: ignore[method-assign] + coordinator._call_json = blocked_review # type: ignore[method-assign] + task = asyncio.create_task( + coordinator._perform_brainstorm_prune_review( # type: ignore[attr-defined] + _request(), + "recursive_brainstorm", + reason="barrier test", + ) + ) + await review_started.wait() + + await coordinator.force_brainstorm() + release_review.set() + await asyncio.wait_for(task, timeout=1) + + self.assertEqual(coordinator._accepted_ideas, ["keep this idea"]) # type: ignore[attr-defined] + self.assertEqual(coordinator.get_state().brainstorm_prune_operations_applied, 0) + async def test_skip_brainstorm_is_consumed_once(self) -> None: old_data_dir = system_config.data_dir with tempfile.TemporaryDirectory() as tmpdir: diff --git a/tests/test_manual_proof_assistant_prewarm.py b/tests/integration/test_manual_proof_assistant_prewarm.py similarity index 52% rename from tests/test_manual_proof_assistant_prewarm.py rename to tests/integration/test_manual_proof_assistant_prewarm.py index 94110c9..bdf424c 100644 --- a/tests/test_manual_proof_assistant_prewarm.py +++ b/tests/integration/test_manual_proof_assistant_prewarm.py @@ -13,9 +13,9 @@ async def test_try_to_prove_refreshes_assistant_before_stage_prompt_preflight_er snapshots = [] class FakeAssistantCoordinator: - async def refresh_now(self, snapshot): + def submit_target(self, snapshot): snapshots.append(snapshot) - return None + return snapshot.stable_hash() async def stop_all(self, **kwargs): self.stop_kwargs = kwargs @@ -78,10 +78,85 @@ async def fake_broadcast(event, payload): self.assertEqual(len(snapshots), 1) self.assertEqual(snapshots[0].workflow_mode, "manual_proof_check") + self.assertEqual(snapshots[0].target_kind, "proof_candidate") self.assertEqual(snapshots[0].workflow_phase, "manual_try_to_prove") + self.assertEqual(snapshots[0].user_prompt, "User prompt") + self.assertEqual(snapshots[0].target_statement, "User prompt") + self.assertEqual(snapshots[0].imports, ["Mathlib"]) self.assertEqual(snapshots[0].source_type, "manual_brainstorm") self.assertEqual(snapshots[0].source_id, "manual_aggregator") + async def test_try_to_prove_does_not_wait_for_assistant_refresh(self) -> None: + old_memory_enabled = system_config.agent_conversation_memory_enabled + system_config.agent_conversation_memory_enabled = True + stage_started = False + + class FakeAssistantCoordinator: + def submit_target(self, snapshot): + return snapshot.stable_hash() + + async def stop_all(self, **kwargs): + return None + + class FakeStage: + async def run_manual(self, **kwargs): + nonlocal stage_started + stage_started = True + + runtime_snapshot = ProofRuntimeConfigSnapshot( + brainstorm=ProofRoleConfigSnapshot( + provider="openrouter", + model_id="proof-model", + context_window=4096, + max_output_tokens=512, + ), + paper=ProofRoleConfigSnapshot( + provider="openrouter", + model_id="proof-model", + context_window=4096, + max_output_tokens=512, + ), + validator=ProofRoleConfigSnapshot( + provider="openrouter", + model_id="validator-model", + context_window=4096, + max_output_tokens=512, + ), + assistant=ProofRoleConfigSnapshot( + provider="openrouter", + model_id="assistant-model", + context_window=4096, + max_output_tokens=512, + ), + ) + + async def fake_resolve_manual_source(request, scoped_proof_database=None): + return "SOURCE CONTENT", "Manual source title", "User prompt" + + async def fake_runtime_snapshot(request=None): + return runtime_snapshot + + async def fake_broadcast(event, payload): + return None + + try: + with ( + mock.patch.object(proofs_route, "assistant_proof_search_coordinator", FakeAssistantCoordinator()), + mock.patch.object(proofs_route, "_resolve_manual_source", new=fake_resolve_manual_source), + mock.patch.object(proofs_route, "_get_runtime_snapshot", new=fake_runtime_snapshot), + mock.patch.object(proofs_route, "websocket") as websocket_mock, + mock.patch.object(proofs_route.autonomous_coordinator, "_proof_verification_stage", FakeStage()), + mock.patch.object(proofs_route.ProofVerificationStage, "release_source", new=mock.AsyncMock()), + ): + websocket_mock.broadcast_event = fake_broadcast + await proofs_route._run_manual_proof_check( + ProofCheckRequest(source_type="brainstorm", source_id="manual_aggregator") + ) + finally: + system_config.agent_conversation_memory_enabled = old_memory_enabled + + self.assertTrue(stage_started) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_moto_launcher.py b/tests/integration/test_moto_launcher.py similarity index 77% rename from tests/test_moto_launcher.py rename to tests/integration/test_moto_launcher.py index ccc9c59..8aa0b8c 100644 --- a/tests/test_moto_launcher.py +++ b/tests/integration/test_moto_launcher.py @@ -9,6 +9,58 @@ import moto_launcher +class BackendReadinessTests(TestCase): + def setUp(self) -> None: + self.service = moto_launcher.LaunchedService( + title="MOTO Backend [test]", + pid=123, + mode="background", + log_path="C:/logs/launcher_backend.log", + ) + + def test_health_poll_returns_healthy_payload(self) -> None: + response = mock.MagicMock() + response.status = 200 + response.read.return_value = b'{"status":"healthy","instance_id":"test"}' + response.__enter__.return_value = response + response.__exit__.return_value = False + with mock.patch.object(moto_launcher, "is_pid_running", return_value=True): + with mock.patch.object(moto_launcher, "urlopen", return_value=response): + payload = moto_launcher.wait_for_backend_health( + "http://localhost:8000", + self.service, + timeout_seconds=1, + poll_interval_seconds=0, + ) + self.assertEqual(payload["status"], "healthy") + + def test_early_exit_reports_backend_log(self) -> None: + with mock.patch.object(moto_launcher, "is_pid_running", return_value=False): + with self.assertRaisesRegex(RuntimeError, "launcher_backend.log"): + moto_launcher.wait_for_backend_health( + "http://localhost:8000", + self.service, + timeout_seconds=1, + poll_interval_seconds=0, + ) + + def test_frontend_early_exit_reports_frontend_log(self) -> None: + frontend = moto_launcher.LaunchedService( + title="MOTO Frontend [test]", + pid=456, + mode="background", + log_path="C:/logs/launcher_frontend.log", + ) + with mock.patch.object(moto_launcher, "is_pid_running", return_value=False): + with self.assertRaisesRegex(RuntimeError, "launcher_frontend.log"): + moto_launcher.wait_for_frontend_ready( + "http://localhost:5173", + frontend, + timeout_seconds=1, + poll_interval_seconds=0, + ) + + class ResolveInstanceRuntimeTests(TestCase): def test_defaults_free_uses_default_instance(self) -> None: with mock.patch.dict(os.environ, {}, clear=True): @@ -433,54 +485,131 @@ def test_check_node_installation_prepends_detected_node_dir_for_npm_scripts(self self.assertEqual(os.environ["PATH"].split(os.pathsep)[0], str(node_dir.resolve())) - def test_frontend_dependency_install_runs_audit_fix_for_clean_git_vulnerabilities(self) -> None: + def test_frontend_dependency_install_uses_lockfile_and_read_only_audit(self) -> None: with tempfile.TemporaryDirectory() as temp_dir: repo_root = Path(temp_dir) - (repo_root / "frontend").mkdir() + frontend_dir = repo_root / "frontend" + frontend_dir.mkdir() + (frontend_dir / "package-lock.json").write_text("{}", encoding="utf-8") - install_result = mock.Mock( - returncode=0, - stdout="added 1 package, and audited 1 package\n1 high severity vulnerability", + install_result = mock.Mock(returncode=0, stdout="added 1 package") + audit_result = mock.Mock(returncode=0, stdout="found 0 vulnerabilities") + + with mock.patch.object(moto_launcher, "SCRIPT_DIR", repo_root): + with mock.patch.object(moto_launcher, "get_npm_command", return_value="npm"): + with mock.patch.object(moto_launcher.subprocess, "run", side_effect=[install_result, audit_result]) as run: + _, vulnerability_warning = moto_launcher.install_frontend_dependencies() + + self.assertFalse(vulnerability_warning) + self.assertEqual(run.call_count, 2) + self.assertEqual(run.call_args_list[0].args[0], ["npm", "ci"]) + self.assertEqual(run.call_args_list[1].args[0], ["npm", "audit", "--audit-level=high"]) + self.assertTrue((frontend_dir / "node_modules" / ".moto_package_lock.sha256").exists()) + + def test_frontend_dependency_install_reconciles_existing_modules_non_destructively(self) -> None: + with tempfile.TemporaryDirectory() as temp_dir: + repo_root = Path(temp_dir) + frontend_dir = repo_root / "frontend" + (frontend_dir / "node_modules").mkdir(parents=True) + (frontend_dir / "package-lock.json").write_text("{}", encoding="utf-8") + + install_result = mock.Mock(returncode=0, stdout="up to date") + audit_result = mock.Mock(returncode=0, stdout="found 0 vulnerabilities") + + with mock.patch.object(moto_launcher, "SCRIPT_DIR", repo_root): + with mock.patch.object(moto_launcher, "get_npm_command", return_value="npm"): + with mock.patch.object(moto_launcher.subprocess, "run", side_effect=[install_result, audit_result]) as run: + _, vulnerability_warning = moto_launcher.install_frontend_dependencies() + + self.assertFalse(vulnerability_warning) + self.assertEqual(run.call_count, 2) + self.assertEqual(run.call_args_list[0].args[0], ["npm", "install", "--package-lock=false", "--no-save"]) + self.assertEqual(run.call_args_list[1].args[0], ["npm", "audit", "--audit-level=high"]) + + def test_frontend_dependency_install_skips_reinstall_when_lock_marker_matches(self) -> None: + with tempfile.TemporaryDirectory() as temp_dir: + repo_root = Path(temp_dir) + frontend_dir = repo_root / "frontend" + node_modules_dir = frontend_dir / "node_modules" + node_modules_dir.mkdir(parents=True) + package_lock = frontend_dir / "package-lock.json" + package_lock.write_text("{}", encoding="utf-8") + (node_modules_dir / ".moto_package_lock.sha256").write_text( + moto_launcher.file_sha256(package_lock), + encoding="utf-8", ) - fix_result = mock.Mock(returncode=0, stdout="fixed 1 vulnerability") + + audit_result = mock.Mock(returncode=0, stdout="found 0 vulnerabilities") with mock.patch.object(moto_launcher, "SCRIPT_DIR", repo_root): with mock.patch.object(moto_launcher, "get_npm_command", return_value="npm"): - with mock.patch.object(moto_launcher.subprocess, "run", side_effect=[install_result, fix_result]) as run: + with mock.patch.object(moto_launcher.subprocess, "run", return_value=audit_result) as run: _, vulnerability_warning = moto_launcher.install_frontend_dependencies() self.assertFalse(vulnerability_warning) + run.assert_called_once() + self.assertEqual(run.call_args.args[0], ["npm", "audit", "--audit-level=high"]) + + def test_frontend_dependency_install_warns_on_high_audit_without_mutating(self) -> None: + with tempfile.TemporaryDirectory() as temp_dir: + repo_root = Path(temp_dir) + frontend_dir = repo_root / "frontend" + frontend_dir.mkdir() + (frontend_dir / "package-lock.json").write_text("{}", encoding="utf-8") + + install_result = mock.Mock(returncode=0, stdout="added 1 package") + audit_result = mock.Mock(returncode=1, stdout="1 high severity vulnerability") + + with mock.patch.object(moto_launcher, "SCRIPT_DIR", repo_root): + with mock.patch.object(moto_launcher, "get_npm_command", return_value="npm"): + with mock.patch.object(moto_launcher.subprocess, "run", side_effect=[install_result, audit_result]) as run: + _, vulnerability_warning = moto_launcher.install_frontend_dependencies() + + self.assertTrue(vulnerability_warning) self.assertEqual(run.call_count, 2) - self.assertEqual(run.call_args_list[0].args[0], ["npm", "install"]) - self.assertEqual(run.call_args_list[1].args[0], ["npm", "audit", "fix"]) + self.assertEqual(run.call_args_list[0].args[0], ["npm", "ci"]) + self.assertEqual(run.call_args_list[1].args[0], ["npm", "audit", "--audit-level=high"]) - def test_frontend_dependency_install_runs_audit_fix_for_npm_remediation_instruction(self) -> None: + def test_frontend_dependency_install_reports_windows_file_lock_with_existing_modules(self) -> None: with tempfile.TemporaryDirectory() as temp_dir: repo_root = Path(temp_dir) - (repo_root / "frontend").mkdir() + frontend_dir = repo_root / "frontend" + (frontend_dir / "node_modules").mkdir(parents=True) + (frontend_dir / "package-lock.json").write_text("{}", encoding="utf-8") install_result = mock.Mock( - returncode=0, - stdout=( - "27 packages are looking for funding\n" - " run `npm fund` for details\n\n" - "2 vulnerabilities (1 moderate, 1 high)\n\n" - "To address all issues, run:\n" - " npm audit fix\n\n" - "Run `npm audit` for details.\n" - ), + returncode=-4048, + stdout="npm error code EPERM\nnpm error syscall unlink\noperation not permitted", ) - fix_result = mock.Mock(returncode=0, stdout="found 0 vulnerabilities") + audit_result = mock.Mock(returncode=0, stdout="found 0 vulnerabilities") + + with mock.patch.object(moto_launcher, "SCRIPT_DIR", repo_root): + with mock.patch.object(moto_launcher, "get_npm_command", return_value="npm"): + with mock.patch.object(moto_launcher.subprocess, "run", side_effect=[install_result, audit_result]) as run: + with mock.patch.object(moto_launcher, "exit_with_pause", side_effect=RuntimeError("exit")): + with self.assertRaisesRegex(RuntimeError, "exit"): + moto_launcher.install_frontend_dependencies() + + run.assert_called_once() + self.assertEqual(run.call_args_list[0].args[0], ["npm", "install", "--package-lock=false", "--no-save"]) + + def test_frontend_dependency_install_falls_back_when_lockfile_missing(self) -> None: + with tempfile.TemporaryDirectory() as temp_dir: + repo_root = Path(temp_dir) + (repo_root / "frontend").mkdir() + + install_result = mock.Mock(returncode=0, stdout="added 1 package") + audit_result = mock.Mock(returncode=0, stdout="found 0 vulnerabilities") with mock.patch.object(moto_launcher, "SCRIPT_DIR", repo_root): with mock.patch.object(moto_launcher, "get_npm_command", return_value="npm"): - with mock.patch.object(moto_launcher.subprocess, "run", side_effect=[install_result, fix_result]) as run: + with mock.patch.object(moto_launcher.subprocess, "run", side_effect=[install_result, audit_result]) as run: _, vulnerability_warning = moto_launcher.install_frontend_dependencies() self.assertFalse(vulnerability_warning) self.assertEqual(run.call_count, 2) self.assertEqual(run.call_args_list[0].args[0], ["npm", "install"]) - self.assertEqual(run.call_args_list[1].args[0], ["npm", "audit", "fix"]) + self.assertEqual(run.call_args_list[1].args[0], ["npm", "audit", "--audit-level=high"]) class LinuxLauncherStrategyTests(TestCase): diff --git a/tests/test_moto_updater.py b/tests/integration/test_moto_updater.py similarity index 90% rename from tests/test_moto_updater.py rename to tests/integration/test_moto_updater.py index 0283f71..fa84e38 100644 --- a/tests/test_moto_updater.py +++ b/tests/integration/test_moto_updater.py @@ -326,6 +326,33 @@ def test_update_available_still_true_for_same_version_new_commit(self) -> None: class LauncherStateTests(TestCase): + def test_nonsecret_json_metadata_rejects_credential_fields(self) -> None: + with tempfile.TemporaryDirectory() as temp_dir: + output_path = Path(temp_dir) / "metadata.json" + + for payload in ( + {"api_key": "example"}, + {"nested": {"refresh_token": "example"}}, + {"instances": [{"authorization": "Bearer example"}]}, + ): + with self.subTest(payload=payload): + with self.assertRaises(ValueError): + moto_updater._write_nonsecret_json_metadata(output_path, payload) + + self.assertFalse(output_path.exists()) + + def test_nonsecret_json_metadata_allows_keyring_namespace_selector(self) -> None: + with tempfile.TemporaryDirectory() as temp_dir: + output_path = Path(temp_dir) / "metadata.json" + payload = { + "instance_id": "instance_one", + "keyring_namespace": "instance_one", + } + + moto_updater._write_nonsecret_json_metadata(output_path, payload) + + self.assertEqual(json.loads(output_path.read_text(encoding="utf-8")), payload) + def test_cleanup_launcher_state_removes_dead_instances(self) -> None: with tempfile.TemporaryDirectory() as temp_dir: state_path = Path(temp_dir) / ".moto_launcher_state.json" @@ -380,6 +407,44 @@ def fake_is_pid_running(pid: int | None) -> bool: ["current", "other"], ) + def test_launcher_state_writer_persists_only_public_runtime_fields(self) -> None: + with tempfile.TemporaryDirectory() as temp_dir: + state_path = Path(temp_dir) / ".moto_launcher_state.json" + payload = { + "instances": [{ + "instance_id": "instance_one", + "backend_window_pid": 100, + "frontend_window_pid": 101, + "backend_port": 8000, + "frontend_port": 5173, + "data_root": "data", + "log_root": "logs", + "storage_prefix": "instance_one", + "unknown_field": "discard-me", + "keyring_namespace": "discard-selector", + }], + "unknown_top_level": "discard-me", + } + + with mock.patch.object(moto_updater, "LAUNCHER_STATE_PATH", state_path): + moto_updater._save_launcher_state(payload) + saved = json.loads(state_path.read_text(encoding="utf-8")) + + self.assertEqual(set(saved), {"instances"}) + self.assertEqual( + set(saved["instances"][0]), + { + "instance_id", + "backend_window_pid", + "frontend_window_pid", + "backend_port", + "frontend_port", + "data_root", + "log_root", + "storage_prefix", + }, + ) + def test_last_instance_record_does_not_persist_keyring_namespace(self) -> None: with tempfile.TemporaryDirectory() as temp_dir: record_path = Path(temp_dir) / ".moto_last_instance.json" @@ -389,7 +454,6 @@ def test_last_instance_record_does_not_persist_keyring_namespace(self) -> None: instance_id="instance_one", data_root="data", log_root="logs", - keyring_namespace="instance_one", storage_prefix="instance_one", ) payload = json.loads(record_path.read_text(encoding="utf-8")) diff --git a/tests/test_oauth_provider_cooldown.py b/tests/integration/test_oauth_provider_cooldown.py similarity index 85% rename from tests/test_oauth_provider_cooldown.py rename to tests/integration/test_oauth_provider_cooldown.py index 179b969..120d588 100644 --- a/tests/test_oauth_provider_cooldown.py +++ b/tests/integration/test_oauth_provider_cooldown.py @@ -8,9 +8,9 @@ RetryableProviderError, ) from backend.shared.config import system_config -from backend.shared.model_error_utils import is_transient_model_call_error +from backend.shared.model_error_utils import is_retryable_model_output_error, is_transient_model_call_error from backend.shared.models import ModelConfig -from backend.shared.openai_codex_client import OAuthUsageLimitError, OpenAICodexClient +from backend.shared.openai_codex_client import OAuthUsageLimitError, OpenAICodexClient, OpenAICodexRequestError from backend.shared.provider_notification_store import record_provider_notification from backend.shared.proof_search.assistant_coordinator import ( _assistant_oauth_provider_is_cooling_down, @@ -119,6 +119,48 @@ async def test_cooldown_lm_fallback_restores_to_codex_after_reset(self) -> None: self.assertEqual(self.manager._role_fallback_state["agg_sub1"], "openai_codex_oauth") self.assertNotIn("agg_sub1", self.manager._oauth_cooldown_fallback_roles) + async def test_codex_output_truncation_bypasses_fallback_and_unrecoverable_notification(self) -> None: + role_id = "autonomous_proof_formalization_brainstorm" + self.manager.configure_role( + role_id, + ModelConfig( + provider="openai_codex_oauth", + model_id="gpt-5.5", + lm_studio_fallback_id="local-fallback", + context_window=128000, + max_output_tokens=8192, + ), + ) + truncation_error = OpenAICodexRequestError( + "OpenAI Codex completion failed: " + '{"type":"response.incomplete","response":{"status":"incomplete",' + '"incomplete_details":{"reason":"max_output_tokens"}}}' + ) + + with ( + mock.patch.object(system_config, "generic_mode", False), + mock.patch( + "backend.shared.api_client_manager.openai_codex_client.generate_completion", + new=mock.AsyncMock(side_effect=truncation_error), + ), + mock.patch.object( + self.manager, + "_broadcast_unrecoverable_codex_error", + new=mock.AsyncMock(), + ) as unrecoverable_broadcast, + ): + with self.assertRaises(OpenAICodexRequestError) as ctx: + await self.manager.generate_completion( + task_id="proof_form_000", + role_id=role_id, + model="gpt-5.5", + messages=[{"role": "user", "content": "prove"}], + ) + + self.assertTrue(is_retryable_model_output_error(ctx.exception)) + self.assertEqual(self.manager._role_fallback_state[role_id], "openai_codex_oauth") + unrecoverable_broadcast.assert_not_awaited() + async def test_wait_for_oauth_provider_cooldown_sleeps_until_expired(self) -> None: self.manager._oauth_provider_cooldowns["openai_codex_oauth"] = { "provider": "openai_codex_oauth", diff --git a/tests/integration/test_proof_routes.py b/tests/integration/test_proof_routes.py new file mode 100644 index 0000000..e8a4ace --- /dev/null +++ b/tests/integration/test_proof_routes.py @@ -0,0 +1,433 @@ +from datetime import datetime +from pathlib import Path +import tempfile +from unittest import IsolatedAsyncioTestCase, TestCase, mock + +from fastapi import FastAPI +from fastapi.testclient import TestClient + +from backend.api.routes import proofs as proofs_route +from backend.autonomous.memory.proof_database import ProofDatabase +from backend.shared.models import ProofCandidate, ProofRecord + + +class ManualProofScopeRouteTests(TestCase): + def setUp(self) -> None: + app = FastAPI() + app.include_router(proofs_route.router) + self.client = TestClient(app) + self._lean_enabled = proofs_route.system_config.lean4_enabled + proofs_route.system_config.lean4_enabled = False + + def tearDown(self) -> None: + proofs_route.system_config.lean4_enabled = self._lean_enabled + + def test_current_manual_proof_listing_uses_manual_database(self) -> None: + manual_db = mock.Mock() + manual_db.get_all_proofs = mock.AsyncMock(return_value=[]) + manual_db.count_proofs.return_value = {"total": 0, "novel": 0, "known": 0} + + with mock.patch.object(proofs_route, "manual_proof_database", manual_db): + response = self.client.get("/api/proofs?scope=manual") + + self.assertEqual(response.status_code, 200) + self.assertEqual(response.json()["scope"], "manual") + manual_db.get_all_proofs.assert_awaited_once_with() + + def test_manual_proof_library_uses_archived_history_only(self) -> None: + manual_db = mock.Mock() + manual_db.list_proof_library_from_history = mock.AsyncMock( + side_effect=[ + [ + { + "proof_id": "proof_history", + "session_id": "manual_proofs_2026-01-01_00-00-00", + "novel": True, + } + ], + [ + { + "proof_id": "proof_history", + "session_id": "manual_proofs_2026-01-01_00-00-00", + "novel": True, + } + ], + ] + ) + + with mock.patch.object(proofs_route, "manual_proof_database", manual_db): + response = self.client.get("/api/proofs/library?scope=manual") + + payload = response.json() + self.assertEqual(response.status_code, 200) + self.assertEqual(payload["scope"], "manual") + self.assertEqual(payload["counts"]["listed"], 1) + self.assertEqual(manual_db.list_proof_library_from_history.await_count, 2) + + def test_proof_library_category_filter_routes_to_database(self) -> None: + proof_db = mock.Mock() + proof_db.list_proof_library = mock.AsyncMock( + side_effect=[ + [ + { + "proof_id": "proof_novel", + "session_id": "session_a", + "novel": True, + "novelty_tier": "mathematical_discovery", + }, + { + "proof_id": "proof_duplicate_novel", + "session_id": "session_a", + "novel": True, + "novelty_tier": "duplicate_novel", + }, + { + "proof_id": "proof_known", + "session_id": "session_a", + "novel": False, + "novelty_tier": "not_novel", + }, + ], + [ + { + "proof_id": "proof_duplicate_novel", + "session_id": "session_a", + "novel": True, + "novelty_tier": "duplicate_novel", + } + ], + ] + ) + + with mock.patch.object(proofs_route, "proof_database", proof_db): + response = self.client.get("/api/proofs/library?category=duplicate_novel") + + payload = response.json() + self.assertEqual(response.status_code, 200) + self.assertEqual(payload["category"], "duplicate_novel") + self.assertEqual(payload["counts"]["total"], 3) + self.assertEqual(payload["counts"]["listed"], 1) + self.assertEqual(payload["counts"]["novel"], 1) + self.assertEqual(payload["counts"]["duplicate_novel"], 1) + self.assertEqual(payload["counts"]["not_novel"], 1) + self.assertEqual( + proof_db.list_proof_library.await_args_list, + [ + mock.call(novel_only=None, category="all"), + mock.call(novel_only=None, category="duplicate_novel"), + ], + ) + + def test_proof_library_category_counts_are_global_for_filtered_tabs(self) -> None: + proof_db = mock.Mock() + proof_db.list_proof_library = mock.AsyncMock( + side_effect=[ + [ + { + "proof_id": "proof_novel", + "session_id": "session_a", + "novel": True, + "novelty_tier": "mathematical_discovery", + }, + { + "proof_id": "proof_duplicate_novel", + "session_id": "session_a", + "novel": True, + "novelty_tier": "duplicate_novel", + }, + ], + [ + { + "proof_id": "proof_novel", + "session_id": "session_a", + "novel": True, + "novelty_tier": "mathematical_discovery", + } + ], + ] + ) + + with mock.patch.object(proofs_route, "proof_database", proof_db): + response = self.client.get("/api/proofs/library?category=novel") + + payload = response.json() + self.assertEqual(response.status_code, 200) + self.assertEqual(payload["counts"]["listed"], 1) + self.assertEqual(payload["counts"]["novel"], 1) + self.assertEqual(payload["counts"]["duplicate_novel"], 1) + self.assertEqual(payload["counts"]["total"], 2) + + def test_proof_library_novel_count_uses_strict_novelty_tiers(self) -> None: + proof_db = mock.Mock() + proof_db.list_proof_library = mock.AsyncMock( + side_effect=[ + [ + { + "proof_id": "proof_legacy_true_not_novel", + "session_id": "session_a", + "novel": True, + "novelty_tier": "not_novel", + }, + { + "proof_id": "proof_unknown_true", + "session_id": "session_a", + "novel": True, + "novelty_tier": "novel", + }, + { + "proof_id": "proof_novel", + "session_id": "session_a", + "novel": True, + "novelty_tier": "novel_variant", + }, + ], + [ + { + "proof_id": "proof_novel", + "session_id": "session_a", + "novel": True, + "novelty_tier": "novel_variant", + }, + ], + ] + ) + + with mock.patch.object(proofs_route, "proof_database", proof_db): + response = self.client.get("/api/proofs/library?category=novel") + + payload = response.json() + self.assertEqual(response.status_code, 200) + self.assertEqual(payload["counts"]["total"], 3) + self.assertEqual(payload["counts"]["listed"], 1) + self.assertEqual(payload["counts"]["novel"], 1) + + def test_proof_library_category_filter_routes_to_manual_database(self) -> None: + manual_db = mock.Mock() + manual_db.list_proof_library_from_history = mock.AsyncMock( + side_effect=[ + [ + { + "proof_id": "proof_novel", + "session_id": "manual_session", + "novel": True, + "novelty_tier": "mathematical_discovery", + }, + { + "proof_id": "proof_duplicate_novel", + "session_id": "manual_session", + "novel": True, + "novelty_tier": "duplicate_novel", + }, + ], + [ + { + "proof_id": "proof_duplicate_novel", + "session_id": "manual_session", + "novel": True, + "novelty_tier": "duplicate_novel", + } + ], + ] + ) + + with mock.patch.object(proofs_route, "manual_proof_database", manual_db): + response = self.client.get("/api/proofs/library?scope=manual&category=duplicate_novel") + + payload = response.json() + self.assertEqual(response.status_code, 200) + self.assertEqual(payload["scope"], "manual") + self.assertEqual(payload["counts"]["total"], 2) + self.assertEqual(payload["counts"]["listed"], 1) + self.assertEqual(payload["counts"]["duplicate_novel"], 1) + self.assertEqual( + manual_db.list_proof_library_from_history.await_args_list, + [ + mock.call( + proofs_route._manual_proof_history_root(), + novel_only=None, + category="all", + ), + mock.call( + proofs_route._manual_proof_history_root(), + novel_only=None, + category="duplicate_novel", + ), + ], + ) + + def test_manual_certificate_routes_use_manual_database(self) -> None: + proof = ProofRecord( + proof_id="manual_proof_1", + theorem_statement="Manual theorem statement.", + theorem_name="manual_theorem", + source_type="brainstorm", + source_id="manual_aggregator", + source_title="Manual Aggregator", + lean_code="theorem manual_theorem : True := by trivial", + solver="Lean 4", + created_at=datetime(2026, 1, 1), + novel=True, + novelty_reasoning="Manual proof route test.", + attempt_count=1, + ) + manual_db = mock.Mock() + manual_db.get_proof = mock.AsyncMock(return_value=proof) + manual_db.get_lean_code = mock.AsyncMock(return_value=proof.lean_code) + + with mock.patch.object(proofs_route, "manual_proof_database", manual_db): + json_response = self.client.get("/api/proofs/manual_proof_1/certificate?scope=manual") + lean_response = self.client.get("/api/proofs/manual_proof_1/certificate.lean?scope=manual") + + self.assertEqual(json_response.status_code, 200) + self.assertEqual(json_response.json()["proof_id"], "manual_proof_1") + self.assertEqual(lean_response.status_code, 200) + self.assertIn("theorem manual_theorem", lean_response.text) + self.assertEqual(manual_db.get_proof.await_count, 2) + self.assertEqual(manual_db.get_lean_code.await_count, 2) + + def test_archived_certificate_normalizes_legacy_mapping_at_response_boundary(self) -> None: + legacy_payload = { + "proof_id": "legacy_archived_proof", + "theorem_statement": "Legacy archived theorem.", + "source_type": "paper", + "source_id": "legacy_paper", + "source_title": "Legacy Paper", + "lean_code": "theorem legacy_archived_theorem : True := by trivial", + "novel": True, + "novelty_reasoning": "Legacy novelty reasoning.", + "created_at": "2025-01-02T03:04:05", + "solver_hints": None, + "dependencies": None, + } + proof_db = mock.Mock() + proof_db.get_library_proof = mock.AsyncMock(return_value=legacy_payload) + + with mock.patch.object(proofs_route, "proof_database", proof_db): + response = self.client.get( + "/api/proofs/library/legacy_session/legacy_archived_proof/certificate" + ) + + payload = response.json() + self.assertEqual(response.status_code, 200) + self.assertEqual(payload["proof_id"], "legacy_archived_proof") + self.assertEqual(payload["run_id"], "legacy:paper:legacy_paper") + self.assertEqual(payload["user_prompt"], "Legacy Paper") + self.assertEqual(payload["novelty_tier"], "novel_formulation") + self.assertEqual(payload["independent_novelty_tier"], "novel_formulation") + self.assertEqual( + payload["independent_novelty_reasoning"], + "Legacy novelty reasoning.", + ) + self.assertEqual(payload["solver_hints"], []) + self.assertEqual(payload["dependencies"], []) + + +class ManualAggregatorProofEventLogTests(IsolatedAsyncioTestCase): + async def test_manual_aggregator_proof_event_is_broadcast_and_persisted_with_same_id(self) -> None: + with ( + mock.patch.object(proofs_route.websocket, "broadcast_event", new=mock.AsyncMock()) as broadcast, + mock.patch.object(proofs_route.event_log, "add_event", new=mock.AsyncMock()) as add_event, + ): + await proofs_route._broadcast_manual_aggregator_proof_event( + "proof_check_complete", + { + "source_type": "brainstorm", + "source_id": "manual_aggregator", + "verified_count": 1, + "novel_count": 0, + }, + ) + + broadcast.assert_awaited_once() + add_event.assert_awaited_once() + _, broadcast_payload = broadcast.await_args.args + _, message, persisted_payload = add_event.await_args.args + + self.assertEqual(message, "Proof check complete: 1 verified, 0 novel") + self.assertEqual( + broadcast_payload["manual_event_id"], + persisted_payload["manual_event_id"], + ) + self.assertEqual(persisted_payload["source_id"], "manual_aggregator") + + +class ProofDatabaseCleanupTests(IsolatedAsyncioTestCase): + async def asyncSetUp(self) -> None: + self._tempdir = tempfile.TemporaryDirectory() + self.base_dir = Path(self._tempdir.name) / "proofs" + self.history_root = Path(self._tempdir.name) / "manual_proof_runs" + self.db = ProofDatabase() + self.db.set_base_dir(self.base_dir) + await self.db.initialize() + + async def asyncTearDown(self) -> None: + self._tempdir.cleanup() + + def _proof_record(self) -> ProofRecord: + return ProofRecord( + proof_id="proof_001", + theorem_statement="Cleanup theorem statement.", + theorem_name="cleanup_theorem", + source_type="brainstorm", + source_id="topic_cleanup", + source_title="Cleanup Topic", + lean_code="theorem cleanup_theorem : True := by trivial", + solver="Lean 4", + created_at=datetime(2026, 1, 1), + novel=True, + novelty_reasoning="Cleanup regression proof.", + attempt_count=1, + ) + + def _candidate(self) -> ProofCandidate: + return ProofCandidate( + theorem_id="failed_cleanup_candidate", + statement="Failed cleanup candidate statement.", + formal_sketch="Try proving by contradiction.", + expected_novelty_tier="mathematical_discovery", + prompt_relevance_rationale="Directly relevant to cleanup regression.", + novelty_rationale="Not a standard known result in this test.", + why_not_standard_known_result="Synthetic test target.", + source_excerpt="Failed source excerpt.", + ) + + async def test_clear_failed_candidates_preserves_verified_proof_files(self) -> None: + await self.db.add_proof(self._proof_record()) + await self.db.record_failed_candidate( + "topic_cleanup", + self._candidate(), + "Lean failed before cleanup.", + ) + + self.assertTrue((self.base_dir / "proof_proof_001.json").exists()) + self.assertTrue((self.base_dir / "proof_proof_001_lean.lean").exists()) + self.assertTrue((self.base_dir / "failed" / "topic_cleanup.json").exists()) + + await self.db.clear_failed_candidates() + + self.assertTrue((self.base_dir / "proof_proof_001.json").exists()) + self.assertTrue((self.base_dir / "proof_proof_001_lean.lean").exists()) + self.assertFalse((self.base_dir / "failed" / "topic_cleanup.json").exists()) + self.assertTrue((self.base_dir / "failed").exists()) + + async def test_archive_current_run_does_not_archive_failed_retry_hints(self) -> None: + await self.db.add_proof(self._proof_record()) + await self.db.record_failed_candidate( + "topic_cleanup", + self._candidate(), + "Lean failed before archive.", + ) + + metadata = await self.db.archive_current_run( + self.history_root, + user_prompt="Manual prompt.", + reason="cleanup_test", + ) + + self.assertIsNotNone(metadata) + archived_proofs = self.history_root / metadata["session_id"] / "proofs" + self.assertTrue((archived_proofs / "proof_proof_001.json").exists()) + self.assertTrue((archived_proofs / "proof_proof_001_lean.lean").exists()) + self.assertFalse((archived_proofs / "failed").exists()) + self.assertTrue((self.base_dir / "failed").exists()) + self.assertFalse((self.base_dir / "failed" / "topic_cleanup.json").exists()) diff --git a/tests/test_proof_search.py b/tests/integration/test_proof_search.py similarity index 67% rename from tests/test_proof_search.py rename to tests/integration/test_proof_search.py index e8e1b5b..a7796a2 100644 --- a/tests/test_proof_search.py +++ b/tests/integration/test_proof_search.py @@ -1,6 +1,7 @@ import asyncio import hashlib import json +import sqlite3 import tempfile import time from pathlib import Path @@ -12,6 +13,7 @@ from backend.api.main import app as full_app from backend.api.routes import proof_search as proof_search_route from backend.api.routes import syntheticlib4 as syntheticlib4_route +from backend.shared.proof_identity import CANONICAL_PROOF_IDENTITY_VERSION from backend.shared.proof_search.indexer import ProofSearchIndexer from backend.shared.proof_search import moto_sources from backend.shared.proof_search.models import ProofSearchRequest, UnifiedProofSearchRecord @@ -21,13 +23,14 @@ SEARCH_LEAN_PROOFS_TOOL_SCHEMA, execute_search_lean_proofs, ) +from backend.shared.proof_search.assistant_models import AssistantProofPack, AssistantProofSupport from backend.shared.config import system_config -from backend.shared.models import ProofCandidate +from backend.shared.models import ProofCandidate, ProofRecord from backend.shared.syntheticlib4_client import SyntheticLib4Client from backend.autonomous.agents import proof_formalization_agent as formalization_module -FIXTURE_DIR = Path(__file__).parent / "fixtures" / "syntheticlib4" +FIXTURE_DIR = Path(__file__).resolve().parents[1] / "fixtures" / "syntheticlib4" def _write_snapshot_fixture(root: Path, *, fingerprint: str, theorem_name: str) -> None: @@ -108,6 +111,35 @@ def test_syntheticlib4_records_normalize_for_search(self) -> None: self.assertEqual(first.external_fingerprint, "sl4_mock_fp_001") self.assertEqual(first.release_id, "stable-2026-06-11") self.assertIn("Finset.sum_congr", first.dependency_names) + self.assertTrue(first.canonical_identity_version) + self.assertTrue(first.canonical_theorem_statement_hash) + + def test_moto_normalization_uses_canonical_identity_hashes(self) -> None: + first = moto_sources.normalize_proof_record( + ProofRecord( + proof_id="proof_001", + theorem_statement="theorem spacing\n: True", + source_type="leanoj_final", + source_id="run_001", + lean_code="\r\ntheorem spacing : True := by trivial\r\n", + ) + ) + second = moto_sources.normalize_proof_record( + ProofRecord( + proof_id="proof_002", + theorem_statement=" theorem spacing : True ", + source_type="leanoj_final", + source_id="run_002", + lean_code="theorem spacing : True := by trivial\n", + ) + ) + + self.assertEqual( + first.canonical_theorem_statement_hash, + second.canonical_theorem_statement_hash, + ) + self.assertEqual(first.canonical_lean_code_hash, second.canonical_lean_code_hash) + self.assertTrue(set(first.dedupe_keys()).intersection(second.dedupe_keys())) def test_fixture_client_hydrates_metadata_only_record(self) -> None: client = SyntheticLib4Client(FIXTURE_DIR) @@ -118,18 +150,27 @@ def test_fixture_client_hydrates_metadata_only_record(self) -> None: self.assertIn("domain_restrict_agree_mock", hydrated["lean_code"]) self.assertEqual(hydrated["lean_code_hash"], "code_hash_013") - def test_client_has_built_in_offline_records_when_test_fixtures_are_absent(self) -> None: - with tempfile.TemporaryDirectory() as temp_dir: - client = SyntheticLib4Client(Path(temp_dir) / "missing-fixtures") - - manifest = client.get_release_manifest() - records = client.load_proof_metadata() - validation = client.validate_local_snapshot() - - self.assertEqual(manifest["contract_version"], "moto-syntheticlib4-v1") - self.assertEqual(len(records), 30) - self.assertEqual(validation["fixture_source"], "built_in") - self.assertTrue(validation["valid"]) + def test_runtime_client_has_no_records_when_snapshot_is_inactive(self) -> None: + old_data_dir = system_config.data_dir + try: + with tempfile.TemporaryDirectory() as temp_dir: + system_config.data_dir = str(Path(temp_dir) / "data") + client = SyntheticLib4Client() + + manifest = client.get_release_manifest() + records = client.load_proof_metadata() + status = client.get_status() + validation = client.validate_local_snapshot() + + self.assertEqual(manifest["contract_version"], "moto-syntheticlib4-v1") + self.assertEqual(manifest["proof_count"], 0) + self.assertEqual(records, []) + self.assertFalse(status["membership_active"]) + self.assertEqual(status["auth_mode"], "inactive") + self.assertFalse(validation["valid"]) + self.assertEqual(validation["proof_count"], 0) + finally: + system_config.data_dir = old_data_dir def test_import_snapshot_directory_activates_data_root_snapshot(self) -> None: old_data_dir = system_config.data_dir @@ -198,13 +239,54 @@ def test_snapshot_import_rejects_unsupported_paths_before_activation(self) -> No records = SyntheticLib4Client().load_proof_metadata() - self.assertEqual(len(records), 30) - self.assertNotEqual(records[0]["fingerprint"], "sl4_imported_fp_unsafe") + self.assertEqual(records, []) finally: system_config.data_dir = old_data_dir class ProofSearchIndexerTests(TestCase): + def test_rebuild_atomically_replaces_incompatible_index(self) -> None: + records = load_syntheticlib4_fixture_records(SyntheticLib4Client(FIXTURE_DIR)) + with tempfile.TemporaryDirectory() as temp_dir: + index_path = Path(temp_dir) / "proof_search.sqlite" + conn = sqlite3.connect(index_path) + try: + conn.execute("CREATE TABLE legacy_marker(value TEXT NOT NULL)") + conn.execute("INSERT INTO legacy_marker(value) VALUES ('preserve-until-swap')") + conn.execute("PRAGMA user_version = 1") + conn.commit() + finally: + conn.close() + + indexer = ProofSearchIndexer(index_path) + self.assertFalse(indexer.is_compatible()) + indexer.rebuild(records) + + self.assertTrue(indexer.is_compatible()) + conn = sqlite3.connect(index_path) + try: + columns = { + row[1] + for row in conn.execute("PRAGMA table_info(proof_records)").fetchall() + } + legacy_table = conn.execute( + "SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'legacy_marker'" + ).fetchone() + canonical_version = conn.execute( + "SELECT value FROM proof_index_metadata WHERE key = ?", + ("canonical_identity_version",), + ).fetchone() + finally: + conn.close() + self.assertIsNone(legacy_table) + self.assertIn("canonical_identity_version", columns) + self.assertIn("canonical_theorem_statement_hash", columns) + self.assertIn("canonical_lean_code_hash", columns) + self.assertEqual(canonical_version[0], CANONICAL_PROOF_IDENTITY_VERSION) + self.assertFalse( + any(index_path.parent.glob(f".{index_path.name}.rebuild-*")) + ) + def test_search_enforces_seven_result_cap_and_dedupes(self) -> None: records = load_syntheticlib4_fixture_records(SyntheticLib4Client(FIXTURE_DIR)) duplicate = records[0].model_copy( @@ -289,6 +371,108 @@ def test_large_index_still_caps_results(self) -> None: self.assertEqual(overview.total_records, 1200) self.assertLessEqual(response.result_count, 7) + def test_exact_identity_neighborhood_scales_past_sqlite_bind_limit(self) -> None: + base = load_syntheticlib4_fixture_records(SyntheticLib4Client(FIXTURE_DIR))[0] + statement_hash = "shared-exact-statement" + code_hash = "shared-exact-code" + records = [] + expected_seed_ids = [] + expected_neighbor_ids = [] + for index in range(1101): + occurrence = { + "corpus": "moto", + "corpus_scope": "history", + "session_id": f"session-{index}", + "run_id": f"run-{index}", + "source_type": "paper", + "source_id": f"paper-{index}", + } + seed_id = f"moto:seed-{index:04d}" + neighbor_id = f"moto:neighbor-{index:04d}" + expected_seed_ids.append(seed_id) + expected_neighbor_ids.append(neighbor_id) + records.extend( + [ + base.model_copy( + update={ + **occurrence, + "search_id": seed_id, + "proof_id": f"seed-{index:04d}", + "external_fingerprint": f"seed-fingerprint-{index:04d}", + "theorem_statement_hash": statement_hash, + "lean_code_hash": code_hash, + "canonical_theorem_statement_hash": statement_hash, + "canonical_lean_code_hash": code_hash, + } + ), + base.model_copy( + update={ + **occurrence, + "search_id": neighbor_id, + "proof_id": f"neighbor-{index:04d}", + "external_fingerprint": f"neighbor-fingerprint-{index:04d}", + "theorem_statement_hash": f"neighbor-statement-{index}", + "lean_code_hash": f"neighbor-code-{index}", + "canonical_theorem_statement_hash": f"neighbor-statement-{index}", + "canonical_lean_code_hash": f"neighbor-code-{index}", + } + ), + ] + ) + records.extend( + [ + base.model_copy( + update={ + "corpus": "moto", + "search_id": "moto:statement-only-decoy", + "proof_id": "statement-only-decoy", + "external_fingerprint": "statement-only-decoy", + "session_id": "decoy-session-a", + "run_id": "decoy-run-a", + "source_type": "paper", + "source_id": "decoy-paper-a", + "theorem_statement_hash": statement_hash, + "lean_code_hash": "wrong-code", + "canonical_theorem_statement_hash": statement_hash, + "canonical_lean_code_hash": "wrong-code", + } + ), + base.model_copy( + update={ + "corpus": "moto", + "search_id": "moto:code-only-decoy", + "proof_id": "code-only-decoy", + "external_fingerprint": "code-only-decoy", + "session_id": "decoy-session-b", + "run_id": "decoy-run-b", + "source_type": "paper", + "source_id": "decoy-paper-b", + "theorem_statement_hash": "wrong-statement", + "lean_code_hash": code_hash, + "canonical_theorem_statement_hash": "wrong-statement", + "canonical_lean_code_hash": code_hash, + } + ), + ] + ) + + with tempfile.TemporaryDirectory() as temp_dir: + indexer = ProofSearchIndexer(Path(temp_dir) / "proof_search.sqlite") + indexer.rebuild(records) + neighborhood = indexer.exact_identity_neighborhood( + theorem_statement_hashes=[statement_hash], + lean_code_hashes=[code_hash], + corpora=["moto"], + limit=5000, + ) + + result_ids = [record.search_id for record in neighborhood] + self.assertEqual(len(result_ids), 2202) + self.assertEqual(result_ids[:1101], expected_seed_ids) + self.assertEqual(result_ids[1101:], expected_neighbor_ids) + self.assertNotIn("moto:statement-only-decoy", result_ids) + self.assertNotIn("moto:code-only-decoy", result_ids) + def test_generated_50k_index_keeps_search_bounded(self) -> None: base_records = load_syntheticlib4_fixture_records(SyntheticLib4Client(FIXTURE_DIR)) generated = [] @@ -398,6 +582,47 @@ async def _get_library_proof(session_id: str, proof_id: str): self.assertIn("theorem example", records[0].lean_code) self.assertIn("Mathlib.Init", records[0].dependency_names) + def test_moto_source_loader_keeps_duplicate_novel_and_not_novel_history(self) -> None: + library_entries = [ + { + "session_id": "session_a", + "proof_id": "proof_duplicate", + "theorem_name": "History.duplicateNovel", + "theorem_statement": "theorem duplicate_novel : True", + "source_type": "paper", + "source_id": "paper_001", + "novel": True, + "novelty_tier": "duplicate_novel", + }, + { + "session_id": "session_a", + "proof_id": "proof_not_novel", + "theorem_name": "History.notNovel", + "theorem_statement": "theorem not_novel : True", + "source_type": "paper", + "source_id": "paper_002", + "novel": False, + "novelty_tier": "not_novel", + }, + ] + + async def _list_library(novel_only: bool = False): + self.assertFalse(novel_only) + return library_entries + + async def _get_library_proof(session_id: str, proof_id: str): + match = next(entry for entry in library_entries if entry["proof_id"] == proof_id) + return {**match, "lean_code": f"theorem {proof_id} : True := by trivial"} + + with mock.patch.object(moto_sources.proof_database, "list_proof_library", _list_library), \ + mock.patch.object(moto_sources.proof_database, "get_library_proof", _get_library_proof): + records = asyncio.run(moto_sources._records_from_autonomous_history()) + + tiers = {record.novelty_tier for record in records} + self.assertIn("duplicate_novel", tiers) + self.assertIn("not_novel", tiers) + self.assertEqual(len(records), 2) + class ProofSearchRouteTests(TestCase): def test_overview_and_search_routes_use_shared_service(self) -> None: @@ -407,7 +632,11 @@ async def _prepare_service(path: Path) -> ProofSearchService: return service with tempfile.TemporaryDirectory() as temp_dir: - service = asyncio.run(_prepare_service(Path(temp_dir) / "proof_search.sqlite")) + service = ProofSearchService( + index_path=Path(temp_dir) / "proof_search.sqlite", + syntheticlib4_source=SyntheticLib4Client(FIXTURE_DIR), + ) + asyncio.run(service.rebuild_index()) app = FastAPI() app.include_router(proof_search_route.router) @@ -430,13 +659,12 @@ async def _prepare_service(path: Path) -> ProofSearchService: self.assertEqual(search.json()["searched_corpora"], ["syntheticlib4"]) def test_proof_detail_route_hydrates_syntheticlib4_record(self) -> None: - async def _prepare_service(path: Path) -> ProofSearchService: - service = ProofSearchService(index_path=path) - await service.rebuild_index() - return service - with tempfile.TemporaryDirectory() as temp_dir: - service = asyncio.run(_prepare_service(Path(temp_dir) / "proof_search.sqlite")) + service = ProofSearchService( + index_path=Path(temp_dir) / "proof_search.sqlite", + syntheticlib4_source=SyntheticLib4Client(FIXTURE_DIR), + ) + asyncio.run(service.rebuild_index()) app = FastAPI() app.include_router(proof_search_route.router) @@ -521,6 +749,39 @@ def test_public_proof_search_rejects_limits_above_result_cap(self) -> None: class ProofSearchFreshnessTests(TestCase): + def test_service_rebuilds_incompatible_derived_index(self) -> None: + record = UnifiedProofSearchRecord( + search_id="moto::proof", + corpus="moto", + source_kind="verified_proof", + proof_id="proof", + theorem_statement="theorem proof : True", + lean_code="theorem proof : True := by trivial", + canonical_uri="moto-proof://proof", + ) + with tempfile.TemporaryDirectory() as temp_dir: + index_path = Path(temp_dir) / "proof_search.sqlite" + indexer = ProofSearchIndexer(index_path) + indexer.rebuild([record]) + conn = indexer._connect() + try: + conn.execute("PRAGMA user_version = 0") + conn.commit() + finally: + conn.close() + + service = ProofSearchService(index_path) + calls = [] + + async def _load_records(): + calls.append(True) + return [record] + + service._load_records = _load_records + asyncio.run(service.overview(include_disabled=True)) + + self.assertEqual(calls, [True]) + def test_service_rebuilds_when_source_files_are_newer_than_index(self) -> None: old_data_dir = system_config.data_dir first_record = UnifiedProofSearchRecord( @@ -591,11 +852,19 @@ async def _prepare_service(path: Path) -> ProofSearchService: return service with tempfile.TemporaryDirectory() as temp_dir: - service = asyncio.run(_prepare_service(Path(temp_dir) / "proof_search.sqlite")) + fixture_client = SyntheticLib4Client(FIXTURE_DIR) + service = ProofSearchService( + index_path=Path(temp_dir) / "proof_search.sqlite", + syntheticlib4_source=fixture_client, + ) + asyncio.run(service.rebuild_index()) app = FastAPI() app.include_router(syntheticlib4_route.router) - with mock.patch.object(syntheticlib4_route, "proof_search_service", service): + with ( + mock.patch.object(syntheticlib4_route, "proof_search_service", service), + mock.patch.object(syntheticlib4_route, "syntheticlib4_client", fixture_client), + ): client = TestClient(app) status = client.get("/api/syntheticlib4/status") releases = client.get("/api/syntheticlib4/releases") @@ -635,13 +904,19 @@ def test_retrieve_batch_and_account_proof_routes(self) -> None: app = FastAPI() app.include_router(syntheticlib4_route.router) client = TestClient(app) - - retrieve = client.post( - "/api/syntheticlib4/retrieve-batch", - json={"limit": 7, "include_full_code": False}, - ) - account = client.get("/api/syntheticlib4/account/proofs") - account_search = client.get("/api/syntheticlib4/account/proofs/search?q=finite") + fixture_client = SyntheticLib4Client(FIXTURE_DIR) + + with mock.patch.object( + syntheticlib4_route, + "syntheticlib4_client", + fixture_client, + ): + retrieve = client.post( + "/api/syntheticlib4/retrieve-batch", + json={"limit": 7, "include_full_code": False}, + ) + account = client.get("/api/syntheticlib4/account/proofs") + account_search = client.get("/api/syntheticlib4/account/proofs/search?q=finite") self.assertEqual(retrieve.status_code, 200) self.assertEqual(len(retrieve.json()["proofs"]), 7) @@ -707,7 +982,11 @@ async def _prepare_service(path: Path) -> ProofSearchService: return service with tempfile.TemporaryDirectory() as temp_dir: - service = asyncio.run(_prepare_service(Path(temp_dir) / "proof_search.sqlite")) + service = ProofSearchService( + index_path=Path(temp_dir) / "proof_search.sqlite", + syntheticlib4_source=SyntheticLib4Client(FIXTURE_DIR), + ) + asyncio.run(service.rebuild_index()) overview = asyncio.run( execute_search_lean_proofs({"action": "overview"}, service=service) ) @@ -885,14 +1164,18 @@ def test_tactic_script_uses_assistant_without_normal_path_in_role_search(self) - class FakeAssistantCoordinator: def __init__(self) -> None: self.snapshots = [] + self.consumed = [] def submit_target(self, snapshot): self.snapshots.append(snapshot) return "target_hash" - def get_latest_pack(self, target_hash=None): + def get_latest_reusable_pack(self): return None + def mark_pack_consumed_by_solver(self, target_hash, *, role_id="", task_id=""): + self.consumed.append((target_hash, role_id, task_id)) + class FakeLeanResult: success = True tactic_error_slice = "" @@ -955,23 +1238,128 @@ async def fail_execute_search_lean_proofs(arguments): self.assertIn("theorem target", lean_code) self.assertEqual(attempts[-1].strategy, "tactic_script") self.assertEqual(fake_lean.last_tactics, ["trivial"]) - self.assertEqual(len(fake_assistant.snapshots), 1) - self.assertEqual(fake_assistant.snapshots[0].target_statement, candidate.statement) + self.assertEqual(fake_assistant.snapshots, []) + self.assertEqual(fake_assistant.consumed, []) self.assertEqual(len(calls), 1) self.assertIsNone(calls[0].get("tools")) self.assertIsNone(calls[0].get("tool_choice")) - def test_compiler_aggregator_formalization_roles_are_manual_assistant_targets(self) -> None: - self.assertEqual( - formalization_module._assistant_workflow_mode_for_role( - "autonomous_proof_formalization_compiler_aggregator" - ), - "manual_proof_check", + def test_full_script_reuses_latest_assistant_pack_without_refreshing_between_attempts(self) -> None: + agent = formalization_module.ProofFormalizationAgent( + model_id="test-model", + context_window=8192, + max_output_tokens=1024, + role_id="test_role", + ) + candidate = ProofCandidate( + theorem_id="candidate_full_001", + statement="theorem target : True", + formal_sketch="Prove True.", + expected_novelty_tier="novel_formulation", + prompt_relevance_rationale="Relevant to the prompt.", + novelty_rationale="A test target.", + why_not_standard_known_result="Fixture-only test.", + ) + pack = AssistantProofPack( + workflow_mode="autonomous", + target_kind="proof_candidate", + target_hash="reused_target_hash", + query_summary="prior helpful proof context", + results=[ + AssistantProofSupport( + search_id="manual:proof_1", + corpus="manual", + corpus_scope="history", + proof_id="proof_1", + theorem_name="Memory.Helper", + theorem_statement="theorem helper : True", + proof_description="A helper proof for True.", + lean_code="import Mathlib\n\ntheorem helper : True := by\n trivial\n", + has_hydrated_code=True, + relevance_reason="Matches the target shape.", + ) + ], ) + original_generate = formalization_module.api_client_manager.generate_completion + original_get_lean = formalization_module.get_lean4_client + original_assistant = formalization_module.assistant_proof_search_coordinator + calls = [] + + class FakeAssistantCoordinator: + def __init__(self) -> None: + self.submitted = [] + self.consumed = [] + + def submit_target(self, snapshot): + self.submitted.append(snapshot) + raise AssertionError("Lean attempts must not refresh Assistant proof memory") + + def get_latest_reusable_pack(self): + return pack + + def mark_pack_consumed_by_solver(self, target_hash, *, role_id="", task_id=""): + self.consumed.append((target_hash, role_id, task_id)) + + class FakeLeanResult: + success = False + error_output = "tactic failed" + goal_states = "unsolved goals" + + class FakeLeanClient: + async def check_proof(self, lean_code, timeout): + return FakeLeanResult() + + fake_assistant = FakeAssistantCoordinator() + + async def fake_generate_completion(**kwargs): + calls.append(kwargs) + return { + "choices": [ + { + "message": { + "role": "assistant", + "content": json.dumps({ + "theorem_name": "target", + "lean_code": "import Mathlib\n\ntheorem target : True := by\n trivial\n", + "reasoning": "Use the retrieved helper pattern.", + }), + } + } + ] + } + + try: + formalization_module.api_client_manager.generate_completion = fake_generate_completion + formalization_module.get_lean4_client = lambda: FakeLeanClient() + formalization_module.assistant_proof_search_coordinator = fake_assistant + success, theorem_name, lean_code, attempts = asyncio.run( + agent.prove_candidate( + user_research_prompt="Prove the target.", + source_type="paper", + theorem_candidate=candidate, + source_content="A source asks for theorem target : True.", + max_attempts=2, + source_title="Assistant full-script test", + ) + ) + finally: + formalization_module.api_client_manager.generate_completion = original_generate + formalization_module.get_lean4_client = original_get_lean + formalization_module.assistant_proof_search_coordinator = original_assistant + + self.assertFalse(success) + self.assertEqual(theorem_name, "target") + self.assertIn("theorem target", lean_code) + self.assertEqual(len(attempts), 2) + self.assertEqual(fake_assistant.submitted, []) self.assertEqual( - formalization_module._assistant_workflow_mode_for_role( - "autonomous_proof_formalization_paper_1" - ), - "autonomous", + [entry[0] for entry in fake_assistant.consumed], + ["reused_target_hash", "reused_target_hash"], ) + self.assertEqual(len(calls), 2) + for call in calls: + prompt = call["messages"][0]["content"] + self.assertIn("ASSISTANT RETRIEVED PROOF SUPPORT", prompt) + self.assertIn("Memory.Helper", prompt) + self.assertIn("theorem helper : True", prompt) diff --git a/tests/test_sakana_fugu.py b/tests/integration/test_sakana_fugu.py similarity index 82% rename from tests/test_sakana_fugu.py rename to tests/integration/test_sakana_fugu.py index bed541b..d44d511 100644 --- a/tests/test_sakana_fugu.py +++ b/tests/integration/test_sakana_fugu.py @@ -8,8 +8,9 @@ from backend.shared.api_client_manager import APIClientManager from backend.shared import secret_store from backend.shared.config import system_config +from backend.shared.model_error_utils import is_retryable_model_output_error from backend.shared.models import ModelConfig -from backend.shared.sakana_fugu_client import SakanaFuguAuthError, SakanaFuguClient +from backend.shared.sakana_fugu_client import SakanaFuguAuthError, SakanaFuguClient, SakanaFuguRequestError class SakanaFuguClientTests(IsolatedAsyncioTestCase): @@ -204,3 +205,47 @@ def fake_response() -> dict: self.assertEqual(tracked_models, ["fugu"]) self.assertEqual(logger_calls[0]["provider"], "sakana_fugu") self.assertEqual(logger_calls[0]["tokens_used"], 5) + + async def test_output_truncation_bypasses_fallback_and_unrecoverable_notification(self) -> None: + manager = APIClientManager() + role_id = "autonomous_proof_formalization_brainstorm" + manager.configure_role( + role_id, + ModelConfig( + provider="sakana_fugu", + model_id="fugu", + lm_studio_fallback_id="local-fallback", + context_window=1000000, + max_output_tokens=100000, + ), + ) + truncation_error = SakanaFuguRequestError( + "Sakana Fugu completion failed: " + '{"type":"response.incomplete","response":{"status":"incomplete",' + '"incomplete_details":{"reason":"max_output_tokens"}}}' + ) + + with ( + mock.patch.object(system_config, "generic_mode", False), + mock.patch.object( + cloud_access_route.sakana_fugu_client, + "generate_completion", + mock.AsyncMock(side_effect=truncation_error), + ), + mock.patch.object( + manager, + "_broadcast_unrecoverable_sakana_fugu_error", + new=mock.AsyncMock(), + ) as unrecoverable_broadcast, + ): + with self.assertRaises(SakanaFuguRequestError) as ctx: + await manager.generate_completion( + task_id="proof_form_000", + role_id=role_id, + model="fugu", + messages=[{"role": "user", "content": "prove"}], + ) + + self.assertTrue(is_retryable_model_output_error(ctx.exception)) + self.assertEqual(manager._role_fallback_state[role_id], "sakana_fugu") + unrecoverable_broadcast.assert_not_awaited() diff --git a/tests/test_update_route.py b/tests/integration/test_update_route.py similarity index 100% rename from tests/test_update_route.py rename to tests/integration/test_update_route.py diff --git a/tests/regressions/__init__.py b/tests/regressions/__init__.py new file mode 100644 index 0000000..1304cf2 --- /dev/null +++ b/tests/regressions/__init__.py @@ -0,0 +1,2 @@ +"""Bug-specific regression tests may be moved here in small batches.""" + diff --git a/tests/regressions/test_api_log_persistence.py b/tests/regressions/test_api_log_persistence.py new file mode 100644 index 0000000..3c99ca2 --- /dev/null +++ b/tests/regressions/test_api_log_persistence.py @@ -0,0 +1,104 @@ +import json +from pathlib import Path + +import pytest + +from backend.autonomous.memory.autonomous_api_logger import AutonomousAPILogger +from backend.shared.boost_logger import BoostLogger +from backend.shared.config import system_config + + +@pytest.fixture +def isolated_loggers(tmp_path, monkeypatch): + monkeypatch.setattr(system_config, "data_dir", tmp_path) + monkeypatch.setattr(system_config, "generic_mode", False) + monkeypatch.setattr(system_config, "api_log_store_full_payloads", True) + + autonomous = AutonomousAPILogger() + autonomous._prepared_root_identity = None + autonomous._volatile_payloads.clear() + boost = BoostLogger() + boost._prepared_root_identity = None + boost._volatile_payloads.clear() + yield autonomous, boost, tmp_path + autonomous._volatile_payloads.clear() + boost._volatile_payloads.clear() + + +@pytest.mark.asyncio +async def test_full_payload_debugging_is_volatile(isolated_loggers): + autonomous, boost, root = isolated_loggers + await autonomous.log_api_call( + task_id="task-1", + role_id="role-1", + model="model", + provider="openrouter", + prompt="password=prompt-secret", + response_content="Authorization: Bearer response-secret", + ) + await boost.log_boost_call( + task_id="boost-1", + role_id="role-1", + model="model", + prompt_preview="safe preview", + response_content="token=boost-response-secret", + ) + + autonomous_disk = (root / "auto_api_log.txt").read_text(encoding="utf-8") + boost_disk = (root / "boost_api_log.txt").read_text(encoding="utf-8") + assert "prompt-secret" not in autonomous_disk + assert "response-secret" not in autonomous_disk + assert "boost-response-secret" not in boost_disk + + [autonomous_detail] = await autonomous.get_logs(include_full=True) + [boost_detail] = await boost.get_logs(include_full=True) + assert autonomous_detail["prompt_full"] == "password=prompt-secret" + assert autonomous_detail["response_full"] == "Authorization: Bearer response-secret" + assert boost_detail["response_full"] == "token=boost-response-secret" + + +@pytest.mark.asyncio +async def test_hosted_mode_ignores_full_payload_debug_flag(isolated_loggers, monkeypatch): + autonomous, boost, root = isolated_loggers + monkeypatch.setattr(system_config, "generic_mode", True) + await autonomous.log_api_call( + task_id="task-2", + role_id="role-2", + model="model", + provider="openrouter", + prompt="password=hosted-prompt-secret", + response_content="Authorization: Bearer hosted-response-secret", + ) + await boost.log_boost_call( + task_id="boost-2", + role_id="role-2", + model="model", + prompt_preview="safe preview", + response_content="token=hosted-boost-secret", + ) + + [autonomous_detail] = await autonomous.get_logs(include_full=True) + [boost_detail] = await boost.get_logs(include_full=True) + assert "prompt_full" not in autonomous_detail + assert "response_full" not in autonomous_detail + assert "response_full" not in boost_detail + assert "hosted-prompt-secret" not in (root / "auto_api_log.txt").read_text(encoding="utf-8") + assert "hosted-boost-secret" not in (root / "boost_api_log.txt").read_text(encoding="utf-8") + + +def test_legacy_full_payloads_are_scrubbed_atomically(isolated_loggers): + autonomous, _, root = isolated_loggers + log_path = Path(root) / "auto_api_log.txt" + log_path.write_text(json.dumps({ + "prompt_full": "password=legacy-secret", + "response_full": "Authorization: Bearer legacy-response", + }) + "\n", encoding="utf-8") + autonomous._prepared_root_identity = None + autonomous._prepare_active_root() + + persisted = log_path.read_text(encoding="utf-8") + assert "legacy-secret" not in persisted + assert "legacy-response" not in persisted + record = json.loads(persisted) + assert record["prompt_size"] == len("password=legacy-secret") + assert record["response_size"] == len("Authorization: Bearer legacy-response") diff --git a/tests/regressions/test_chroma_native_lifecycle.py b/tests/regressions/test_chroma_native_lifecycle.py new file mode 100644 index 0000000..4839aef --- /dev/null +++ b/tests/regressions/test_chroma_native_lifecycle.py @@ -0,0 +1,137 @@ +from __future__ import annotations + +import os +from pathlib import Path +import subprocess +import sys +import textwrap + +import pytest + + +@pytest.mark.skipif(os.name != "nt", reason="Windows native-binding release blocker") +def test_real_chroma_rebuild_reopen_survives_subprocess(tmp_path: Path) -> None: + """A native access violation becomes a deterministic nonzero subprocess exit.""" + script = textwrap.dedent( + """ + import asyncio + from pathlib import Path + + from backend.aggregator.core.rag_manager import RAGManager + from backend.shared.config import bind_runtime_roots + + root = Path(r"{root}") + bind_runtime_roots(data_root=root / "data", logs_root=root / "logs") + + async def main(): + manager = RAGManager() + await manager.ensure_initialized() + for collection in manager.collections.values(): + collection.upsert( + ids=["first"], + embeddings=[[0.0, 1.0, 0.0]], + documents=["native lifecycle"], + metadatas=[{{"source_file": "source.txt"}}], + ) + await manager.remove_document("source.txt") + assert all(collection.count() == 0 for collection in manager.collections.values()) + await manager.close() + await manager.ensure_initialized() + assert all(collection.count() == 0 for collection in manager.collections.values()) + await manager.close() + + asyncio.run(main()) + """ + ).format(root=str(tmp_path).replace("\\", "\\\\")) + result = subprocess.run( + [sys.executable, "-c", script], + cwd=Path(__file__).resolve().parents[2], + capture_output=True, + text=True, + timeout=90, + check=False, + ) + assert result.returncode == 0, ( + f"native Chroma subprocess exited {result.returncode}\n" + f"stdout:\n{result.stdout}\nstderr:\n{result.stderr}" + ) + + +@pytest.mark.skipif(os.name != "nt", reason="Windows process-generation cache contract") +def test_windows_process_generation_never_opens_prior_cache(tmp_path: Path) -> None: + """A new backend generation replaces the old cache before native open.""" + root = str(tmp_path).replace("\\", "\\\\") + create_script = textwrap.dedent( + f""" + import asyncio + from pathlib import Path + from backend.aggregator.core.rag_manager import RAGManager + from backend.shared.config import bind_runtime_roots + + root = Path(r"{root}") + bind_runtime_roots(data_root=root / "data", logs_root=root / "logs") + + async def main(): + manager = RAGManager() + await manager.ensure_initialized() + collection = manager.collections[256] + collection.upsert( + ids=["prior"], + embeddings=[[0.0, 1.0, 0.0]], + documents=["prior generation"], + metadatas=[{{"source_file": "prior.txt"}}], + ) + assert collection.count() == 1 + await manager.close() + + asyncio.run(main()) + """ + ) + first = subprocess.run( + [sys.executable, "-c", create_script], + cwd=Path(__file__).resolve().parents[2], + capture_output=True, + text=True, + timeout=90, + check=False, + ) + assert first.returncode == 0, first.stderr + + reopen_script = textwrap.dedent( + f""" + import asyncio + from pathlib import Path + from backend.aggregator.core.rag_manager import RAGManager + from backend.shared.config import bind_runtime_roots + + root = Path(r"{root}") + bind_runtime_roots(data_root=root / "data", logs_root=root / "logs") + + async def main(): + manager = RAGManager() + await manager.prepare_process_generation_cache() + assert manager.collections[256].count() == 0 + manager.collections[256].upsert( + ids=["current"], + embeddings=[[1.0, 0.0, 0.0]], + documents=["current generation"], + metadatas=[{{"source_file": "current.txt"}}], + ) + assert manager.collections[256].count() == 1 + await manager.close() + + asyncio.run(main()) + """ + ) + second = subprocess.run( + [sys.executable, "-c", reopen_script], + cwd=Path(__file__).resolve().parents[2], + capture_output=True, + text=True, + timeout=90, + check=False, + ) + assert second.returncode == 0, ( + f"replacement subprocess exited {second.returncode}\n" + f"stdout:\n{second.stdout}\nstderr:\n{second.stderr}" + ) diff --git a/tests/test_codeql_path_hardening.py b/tests/regressions/test_codeql_path_hardening.py similarity index 60% rename from tests/test_codeql_path_hardening.py rename to tests/regressions/test_codeql_path_hardening.py index 71dcbc8..05db3fb 100644 --- a/tests/test_codeql_path_hardening.py +++ b/tests/regressions/test_codeql_path_hardening.py @@ -8,12 +8,15 @@ from backend.aggregator.core.coordinator import _resolve_uploaded_user_file from backend.aggregator.ingestion.pipeline import IngestionPipeline +from backend.api.routes.aggregator import _clear_uploaded_files, _delete_uploaded_file from backend.api.routes import autonomous as autonomous_route from backend.autonomous.memory.autonomous_rejection_logs import AutonomousRejectionLogs from backend.autonomous.memory.brainstorm_memory import BrainstormMemory from backend.autonomous.memory.paper_library import PaperLibrary +from backend.autonomous.memory.proof_database import ProofDatabase from backend.shared.config import system_config -from backend.shared.models import PaperMetadata +from backend.shared.models import PaperMetadata, ProofRecord +from backend.shared.path_safety import resolve_filename_within_root class IngestionPathHardeningTests(TestCase): @@ -98,6 +101,43 @@ def test_uploaded_user_file_allows_trusted_context_files_only_when_enabled(self) trusted_file.resolve(), ) + def test_delete_uploaded_file_rejects_traversal_and_deletes_lean(self) -> None: + async def run_case() -> None: + with tempfile.TemporaryDirectory() as temp_dir: + uploads = Path(temp_dir) / "uploads" + uploads.mkdir() + lean_file = uploads / "helper.lean" + lean_file.write_text("theorem helper : True := by trivial", encoding="utf-8") + + with mock.patch.object(system_config, "user_uploads_dir", str(uploads)): + self.assertTrue(await _delete_uploaded_file("helper.lean")) + self.assertFalse(lean_file.exists()) + with self.assertRaises(ValueError): + await _delete_uploaded_file("../outside.lean") + with self.assertRaises(ValueError): + await _delete_uploaded_file("not_allowed.md") + + asyncio.run(run_case()) + + def test_clear_uploaded_files_only_deletes_managed_text_uploads(self) -> None: + async def run_case() -> None: + with tempfile.TemporaryDirectory() as temp_dir: + uploads = Path(temp_dir) / "uploads" + uploads.mkdir() + (uploads / "notes.txt").write_text("notes", encoding="utf-8") + (uploads / "helper.lean").write_text("theorem helper : True := by trivial", encoding="utf-8") + (uploads / "keep.bin").write_bytes(b"\x00\x01") + + with mock.patch.object(system_config, "user_uploads_dir", str(uploads)): + deleted = await _clear_uploaded_files() + + self.assertEqual(deleted, 2) + self.assertFalse((uploads / "notes.txt").exists()) + self.assertFalse((uploads / "helper.lean").exists()) + self.assertTrue((uploads / "keep.bin").exists()) + + asyncio.run(run_case()) + class PaperLibraryPathHardeningTests(TestCase): def _library_for(self, base_dir: Path) -> PaperLibrary: @@ -140,6 +180,102 @@ async def run_case() -> None: asyncio.run(run_case()) +class ProofDatabasePathHardeningTests(TestCase): + def _database_for(self, base_dir: Path) -> ProofDatabase: + database = ProofDatabase() + database.set_base_dir(base_dir) + base_dir.mkdir(parents=True, exist_ok=True) + return database + + def test_proof_paths_reject_traversal_and_both_separator_styles(self) -> None: + with tempfile.TemporaryDirectory() as temp_dir: + database = self._database_for(Path(temp_dir) / "proofs") + + for proof_id in ("../evil", r"..\evil", "a/b", r"a\b", ".", ".."): + with self.subTest(proof_id=proof_id): + with self.assertRaises(ValueError): + database._get_record_path(proof_id) + with self.assertRaises(ValueError): + database._get_lean_path(proof_id) + + def test_filename_resolver_rejects_absolute_and_drive_like_paths(self) -> None: + with tempfile.TemporaryDirectory() as temp_dir: + root = Path(temp_dir) / "proofs" + root.mkdir() + for filename in ("/tmp/evil", r"C:\temp\evil", "../evil", r"..\evil"): + with self.subTest(filename=filename): + with self.assertRaises(ValueError): + resolve_filename_within_root(root, filename) + self.assertEqual( + resolve_filename_within_root(root, "proof_001.json"), + (root / "proof_001.json").resolve(), + ) + + def test_proof_paths_remain_inside_active_store(self) -> None: + with tempfile.TemporaryDirectory() as temp_dir: + base_dir = Path(temp_dir) / "proofs" + database = self._database_for(base_dir) + + self.assertEqual( + database._get_record_path("proof_001"), + (base_dir / "proof_proof_001.json").resolve(), + ) + self.assertEqual( + database._get_lean_path("proof_001"), + (base_dir / "proof_proof_001_lean.lean").resolve(), + ) + self.assertEqual( + database._get_failed_candidates_path("topic_001"), + (base_dir / "failed" / "topic_001.json").resolve(), + ) + + def test_failed_candidate_path_rejects_traversal(self) -> None: + with tempfile.TemporaryDirectory() as temp_dir: + database = self._database_for(Path(temp_dir) / "proofs") + + for brainstorm_id in ("../evil", r"..\evil", "a/b", r"a\b", ".", ".."): + with self.subTest(brainstorm_id=brainstorm_id): + with self.assertRaises(ValueError): + database._get_failed_candidates_path(brainstorm_id) + + def test_occurrence_write_rejects_malicious_id_without_creating_artifacts(self) -> None: + async def run_case() -> None: + with tempfile.TemporaryDirectory() as temp_dir: + base_dir = Path(temp_dir) / "proofs" + database = self._database_for(base_dir) + record = ProofRecord( + proof_id="../outside", + theorem_statement="True", + source_type="paper", + source_id="paper_1", + lean_code="theorem safe : True := by trivial", + ) + with self.assertRaises(ValueError): + await database.add_proof_occurrence(record) + self.assertFalse((Path(temp_dir) / "proof_outside.json").exists()) + self.assertEqual(list(base_dir.glob("proof_*")), []) + + asyncio.run(run_case()) + + def test_occurrence_write_creates_both_valid_artifacts(self) -> None: + async def run_case() -> None: + with tempfile.TemporaryDirectory() as temp_dir: + base_dir = Path(temp_dir) / "proofs" + database = self._database_for(base_dir) + record = ProofRecord( + proof_id="proof_custom", + theorem_statement="True", + source_type="paper", + source_id="paper_1", + lean_code="theorem safe : True := by trivial", + ) + await database.add_proof_occurrence(record) + self.assertTrue((base_dir / "proof_proof_custom.json").exists()) + self.assertTrue((base_dir / "proof_proof_custom_lean.lean").exists()) + + asyncio.run(run_case()) + + class BrainstormMemoryPathHardeningTests(TestCase): def _memory_for(self, base_dir: Path) -> BrainstormMemory: memory = BrainstormMemory() diff --git a/tests/regressions/test_context_overflow_emitters.py b/tests/regressions/test_context_overflow_emitters.py new file mode 100644 index 0000000..9bd2eba --- /dev/null +++ b/tests/regressions/test_context_overflow_emitters.py @@ -0,0 +1,316 @@ +import pytest + +from backend.aggregator.core.context_allocator import ContextAllocationError +from backend.aggregator.core.coordinator import Coordinator +from backend.autonomous.core.autonomous_coordinator import AutonomousCoordinator +from backend.autonomous.memory.research_metadata import research_metadata +from backend.compiler.core.compiler_coordinator import CompilerCoordinator +from backend.leanoj.core.leanoj_coordinator import LeanOJConfigurationError, LeanOJCoordinator +from backend.shared.models import ModelConfig +from backend.shared.provider_errors import ProviderContextLengthError, ProviderRouteIdentity + + +def _config(model: str, provider: str = "openrouter") -> ModelConfig: + return ModelConfig( + model_id=model, + provider=provider, + context_window=8192, + max_output_tokens=1024, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("role_id", "configured_role", "model"), + [ + ("aggregator_submitter_1", "aggregator_submitter_1", "submitter-model"), + ("aggregator_validator", "aggregator_validator", "validator-model"), + ("aggregator_single_model", "aggregator_validator", "shared-model"), + ], +) +async def test_aggregator_context_overflow_maps_roles_and_persists_metadata( + monkeypatch, role_id, configured_role, model +): + coordinator = Coordinator() + coordinator.is_running = True + emitted = [] + persisted = [] + + monkeypatch.setattr( + "backend.aggregator.core.coordinator.api_client_manager.get_role_config", + lambda requested_role: _config(model) if requested_role == configured_role else None, + ) + + async def capture(event_type, payload): + emitted.append((event_type, payload)) + + async def capture_persisted(event_type, message, metadata): + persisted.append((event_type, message, metadata)) + + async def clear_queue(): + return None + + coordinator.websocket_broadcaster = capture + monkeypatch.setattr(coordinator, "_add_persisted_event", capture_persisted) + monkeypatch.setattr("backend.aggregator.core.coordinator.queue_manager.clear", clear_queue) + + await coordinator._handle_context_overflow( + ContextAllocationError( + "mandatory direct context overflow", + required_tokens=9000, + available_tokens=7000, + context_window=8192, + output_reserve=1024, + ), + role_id=role_id, + ) + + payload = emitted[0][1] + assert emitted[0][0] == "context_overflow_error" + assert payload["workflow_mode"] == "aggregator" + assert payload["role_id"] == role_id + assert payload["configured_model"] == model + assert payload["configured_provider"] == "openrouter" + assert payload["required_tokens"] == 9000 + assert persisted[0][2] == payload + assert coordinator.fatal_error_payload == payload + + +@pytest.mark.asyncio +async def test_aggregator_child_emits_autonomous_mode_without_manual_persistence(monkeypatch): + coordinator = Coordinator() + coordinator.persist_event_log = False + coordinator.is_running = True + emitted = [] + + monkeypatch.setattr( + "backend.aggregator.core.coordinator.api_client_manager.get_role_config", + lambda _role: _config("child-model"), + ) + + async def clear_queue(): + return None + + coordinator.websocket_broadcaster = lambda event, payload: _capture(emitted, event, payload) + monkeypatch.setattr("backend.aggregator.core.coordinator.queue_manager.clear", clear_queue) + + await coordinator._handle_context_overflow( + ContextAllocationError("context overflow"), + role_id="aggregator_submitter_1", + ) + + assert emitted[0][1]["workflow_mode"] == "autonomous" + + +@pytest.mark.asyncio +async def test_aggregator_context_allocation_error_preserves_provider_route(monkeypatch): + coordinator = Coordinator() + coordinator.is_running = True + emitted = [] + route = ProviderRouteIdentity( + provider="openrouter", + model="effective-model", + role_id="aggregator_submitter_1", + task_id="agg_sub1_001", + host_provider="effective-host", + route_kind="free_rotation", + configured_provider="openrouter", + configured_model="configured-model", + ) + provider_error = ProviderContextLengthError("provider context limit", route=route) + error = ContextAllocationError.from_provider_error( + provider_error, + "submitter provider context mismatch", + required_tokens=9000, + available_tokens=7000, + ) + + async def clear_queue(): + return None + + coordinator.websocket_broadcaster = lambda event, payload: _capture(emitted, event, payload) + monkeypatch.setattr("backend.aggregator.core.coordinator.queue_manager.clear", clear_queue) + monkeypatch.setattr( + "backend.aggregator.core.coordinator.api_client_manager.get_role_config", + lambda _role: _config("configured-model"), + ) + + await coordinator._handle_context_overflow(error, role_id="aggregator_submitter_1") + + payload = emitted[0][1] + assert payload["configured_model"] == "configured-model" + assert payload["effective_model"] == "effective-model" + assert payload["effective_provider"] == "openrouter" + assert payload["effective_host_provider"] == "effective-host" + assert payload["route_kind"] == "free_rotation" + + +async def _capture(target, event, payload): + target.append((event, payload)) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("role_id", "configured_role", "model"), + [ + ("compiler_writer", "compiler_writer", "writer-model"), + ("compiler_validator", "compiler_validator", "validator-model"), + ("compiler_rigor", "compiler_high_param", "rigor-model"), + ], +) +async def test_compiler_context_overflow_maps_roles(monkeypatch, role_id, configured_role, model): + coordinator = CompilerCoordinator() + emitted = [] + coordinator.websocket_broadcaster = lambda event, payload: _capture(emitted, event, payload) + monkeypatch.setattr( + "backend.compiler.core.compiler_coordinator.api_client_manager.get_role_config", + lambda requested_role: _config(model) if requested_role == configured_role else None, + ) + + await coordinator._handle_context_overflow( + RuntimeError("prompt context overflow"), + role_id=role_id, + mode="review", + ) + + payload = emitted[0][1] + assert payload["workflow_mode"] == "compiler" + assert payload["role_id"] == role_id + assert payload["configured_model"] == model + assert payload["mode"] == "review" + assert coordinator.fatal_error_payload == payload + + +@pytest.mark.asyncio +async def test_autonomous_compiler_marks_child_payload_for_parent(monkeypatch): + coordinator = CompilerCoordinator() + coordinator.enable_autonomous_mode() + emitted = [] + coordinator.websocket_broadcaster = lambda event, payload: _capture(emitted, event, payload) + monkeypatch.setattr( + "backend.compiler.core.compiler_coordinator.api_client_manager.get_role_config", + lambda _role: _config("autonomous-writer"), + ) + + await coordinator._handle_context_overflow( + RuntimeError("context overflow"), + role_id="compiler_writer", + ) + + assert emitted[0][1]["workflow_mode"] == "autonomous" + assert coordinator.fatal_error_payload["configured_model"] == "autonomous-writer" + + +@pytest.mark.asyncio +async def test_leanoj_overflow_preserves_phase_role_and_terminal_payload(monkeypatch): + coordinator = LeanOJCoordinator() + coordinator._state.phase = "final_solver" + events = [] + monkeypatch.setattr( + "backend.leanoj.core.leanoj_coordinator.api_client_manager.get_role_config", + lambda role: _config("final-model") if role == "leanoj_final_solver" else None, + ) + + async def persist_and_broadcast(event, payload=None): + events.append((event, payload or {})) + + monkeypatch.setattr(coordinator, "_persist_and_broadcast", persist_and_broadcast) + await coordinator._handle_context_overflow_stop( + RuntimeError("context overflow"), + role_id="leanoj_final_solver", + ) + + payload = events[0][1] + assert payload["workflow_mode"] == "leanoj" + assert payload["phase"] == "final_solver" + assert payload["role_id"] == "leanoj_final_solver" + assert payload["configured_model"] == "final-model" + terminal_payload = { + **coordinator.get_status(), + **coordinator._fatal_stop_payload, + "reason": coordinator._fatal_stop_reason, + "message": coordinator._fatal_stop_message, + } + assert terminal_payload["configured_model"] == "final-model" + assert terminal_payload["phase"] == "final_solver" + + +@pytest.mark.asyncio +async def test_leanoj_wrapped_provider_overflow_is_idempotent_and_preserves_route(monkeypatch): + coordinator = LeanOJCoordinator() + coordinator._state.phase = "initial_brainstorm" + events = [] + route = ProviderRouteIdentity( + provider="openrouter", + model="effective-model", + role_id="leanoj_brainstorm_validator", + task_id="leanoj_brainstorm_val_001", + host_provider="effective-host", + route_kind="boost", + configured_provider="openrouter", + configured_model="configured-model", + ) + cause = ProviderContextLengthError("provider context limit", route=route) + error = LeanOJConfigurationError( + "Proof Solver provider context overflow", + route=route, + ) + error.__cause__ = cause + + async def persist_and_broadcast(event, payload=None): + events.append((event, payload or {})) + + monkeypatch.setattr(coordinator, "_persist_and_broadcast", persist_and_broadcast) + monkeypatch.setattr( + "backend.leanoj.core.leanoj_coordinator.api_client_manager.get_role_config", + lambda _role: _config("configured-model"), + ) + + assert coordinator._is_context_overflow_exception(error) + await coordinator._handle_context_overflow_stop(error) + await coordinator._handle_context_overflow_stop(error) + + overflow_events = [item for item in events if item[0] == "context_overflow_error"] + assert len(overflow_events) == 1 + payload = overflow_events[0][1] + assert payload["role_id"] == "leanoj_brainstorm_validator" + assert payload["configured_model"] == "configured-model" + assert payload["effective_model"] == "effective-model" + assert payload["effective_host_provider"] == "effective-host" + assert payload["route_kind"] == "boost" + terminal_payload = {**coordinator.get_status(), **coordinator._fatal_stop_payload} + assert terminal_payload["phase"] == "initial_brainstorm" + assert terminal_payload["effective_model"] == "effective-model" + + +@pytest.mark.asyncio +async def test_autonomous_parent_propagates_once_and_stale_payload_can_be_reset(monkeypatch): + coordinator = AutonomousCoordinator() + events = [] + coordinator.set_broadcast_callback(lambda event, payload: _capture(events, event, payload)) + + async def stats(): + return {} + + monkeypatch.setattr(research_metadata, "get_stats", stats) + payload = { + "workflow_mode": "autonomous", + "role_id": "compiler_writer", + "configured_model": "child-writer", + } + coordinator._mark_context_overflow_stop(payload) + await coordinator._broadcast_stopped_once() + await coordinator._broadcast_stopped_once() + + assert len(events) == 1 + assert events[0][0] == "auto_research_stopped" + assert events[0][1]["reason"] == "context_overflow" + assert events[0][1]["configured_model"] == "child-writer" + + coordinator._fatal_stop_reason = None + coordinator._fatal_stop_message = "" + coordinator._fatal_stop_payload = {} + coordinator._stop_broadcast_sent = False + await coordinator._broadcast_stopped_once() + assert "configured_model" not in events[1][1] + assert "reason" not in events[1][1] diff --git a/tests/regressions/test_context_overflow_metadata.py b/tests/regressions/test_context_overflow_metadata.py new file mode 100644 index 0000000..0163980 --- /dev/null +++ b/tests/regressions/test_context_overflow_metadata.py @@ -0,0 +1,17 @@ +from backend.shared.context_overflow import context_overflow_model_payload +from backend.shared.models import ModelConfig + + +def test_context_overflow_model_payload_exposes_configured_identity() -> None: + payload = context_overflow_model_payload( + ModelConfig(provider="openrouter", model_id="example/model") + ) + + assert payload == { + "configured_model": "example/model", + "configured_provider": "openrouter", + } + + +def test_context_overflow_model_payload_allows_missing_config() -> None: + assert context_overflow_model_payload(None) == {} diff --git a/tests/test_output_sabotage_regressions.py b/tests/regressions/test_output_sabotage_regressions.py similarity index 96% rename from tests/test_output_sabotage_regressions.py rename to tests/regressions/test_output_sabotage_regressions.py index 1282579..de39fe7 100644 --- a/tests/test_output_sabotage_regressions.py +++ b/tests/regressions/test_output_sabotage_regressions.py @@ -4,7 +4,6 @@ from backend.aggregator.core import context_allocator as context_allocator_module from backend.aggregator.core.rag_manager import rag_manager from backend.compiler.agents.high_param_submitter import HighParamSubmitter -from backend.autonomous.agents.proof_formalization_agent import _assistant_workflow_mode_for_role from backend.shared.config import system_config from backend.shared.models import ContextPack from backend.shared.openrouter_client import CreditExhaustionError @@ -116,10 +115,6 @@ def test_rigor_proof_source_preserves_full_source_context(monkeypatch): assert "direct source context truncated" not in proof_source -def test_compiler_rigor_formalization_uses_compiler_assistant_workflow(): - assert _assistant_workflow_mode_for_role("compiler_rigor_formalization") == "compiler" - - @pytest.mark.asyncio async def test_rigor_llm_provider_failure_is_not_converted_to_decline(monkeypatch): monkeypatch.setattr(system_config, "compiler_high_param_context_window", 10000) diff --git a/tests/regressions/test_proof_context_overflow_metadata.py b/tests/regressions/test_proof_context_overflow_metadata.py new file mode 100644 index 0000000..176c01d --- /dev/null +++ b/tests/regressions/test_proof_context_overflow_metadata.py @@ -0,0 +1,294 @@ +from unittest.mock import AsyncMock + +import pytest + +from backend.autonomous.agents import proof_formalization_agent as formalization_module +from backend.autonomous.agents.proof_formalization_agent import ( + ProofFormalizationAgent, + ProofFormalizationContextOverflowError, +) +from backend.autonomous.core.proof_verification_stage import ( + ProofVerificationStage, + _LeanVerificationOutcome, +) +from backend.compiler.core.compiler_coordinator import CompilerCoordinator +from backend.shared.models import ProofAttemptFeedback, ProofCandidate +from backend.shared.provider_errors import ProviderContextLengthError, ProviderRouteIdentity + + +def _candidate() -> ProofCandidate: + return ProofCandidate(theorem_id="candidate-1", statement="True") + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("method_name", "strategy"), + [ + ("prove_candidate", "full_script"), + ("prove_candidate_tactic_script", "tactic_script"), + ], +) +async def test_typed_provider_overflow_route_survives_formalization( + monkeypatch, + method_name, + strategy, +): + route = ProviderRouteIdentity( + provider="lm_studio", + model="fallback-model:2", + configured_model="configured/model", + configured_provider="openrouter", + ) + monkeypatch.setattr( + formalization_module.api_client_manager, + "generate_completion", + AsyncMock( + side_effect=ProviderContextLengthError( + "context_length_exceeded", + route=route, + ) + ), + ) + + agent = ProofFormalizationAgent( + model_id="configured/model", + context_window=32_000, + max_output_tokens=2_000, + role_id="autonomous_proof_formalization_paper", + ) + method = getattr(agent, method_name) + success, _, _, attempts = await method( + user_research_prompt="Prove the claim.", + source_type="paper", + theorem_candidate=_candidate(), + source_content="A short source.", + max_attempts=1, + ) + + assert success is False + assert len(attempts) == 1 + feedback = attempts[0] + assert feedback.strategy == strategy + assert feedback.configured_model == "configured/model" + assert feedback.configured_provider == "openrouter" + assert feedback.effective_model == "fallback-model:2" + assert feedback.effective_provider == "lm_studio" + assert feedback.overflow_origin == "provider" + assert feedback.prompt_tokens is None + assert feedback.max_input_tokens is None + assert "context_length_exceeded" in feedback.error_output + + +def test_proof_overflow_workflow_scope_is_explicit(): + assert ProofVerificationStage._proof_workflow_mode("automatic") == "autonomous" + assert ProofVerificationStage._proof_workflow_mode("manual") == "manual_proof_check" + assert ProofVerificationStage._proof_workflow_mode("manual_compiler_save") == "compiler" + assert ProofVerificationStage._proof_workflow_mode("manual_compiler_aggregator") == "aggregator" + + +@pytest.mark.asyncio +async def test_proof_overflow_event_is_scoped_nonfatal_and_uses_effective_route(monkeypatch): + feedback = ProofAttemptFeedback( + attempt=1, + theorem_id="candidate-1", + error_output="MANDATORY FULL SOURCE CONTEXT OVERFLOW", + configured_model="configured/model", + configured_provider="openrouter", + effective_model="fallback-model:2", + effective_provider="lm_studio", + overflow_origin="provider", + ) + + async def fake_prove(self, *args, attempt_callback=None, **kwargs): + await attempt_callback(feedback) + return False, "", "", [feedback] + + monkeypatch.setattr(ProofFormalizationAgent, "prove_candidate", fake_prove) + stage = ProofVerificationStage() + monkeypatch.setattr(stage, "_prepare_candidate", AsyncMock(return_value=_candidate())) + monkeypatch.setattr(stage, "_run_smt_check", AsyncMock(return_value=None)) + broadcast = AsyncMock() + + await stage._run_lean_pipeline_for_candidate( + theorem_candidate=_candidate(), + base_event={"source_type": "paper", "source_id": "paper-1"}, + proof_label="A", + user_prompt="Prove the claim.", + source_type="paper", + source_id="paper-1", + source_content="A short source.", + source_title="Paper", + submitter_model="configured/model", + submitter_context=32_000, + submitter_max_tokens=2_000, + role_suffix="paper", + trigger="manual_compiler_save", + novel_proofs_db=None, + broadcast_fn=broadcast, + ) + + overflow_calls = [ + call for call in broadcast.await_args_list if call.args[0] == "proof_context_overflow" + ] + assert len(overflow_calls) == 1 + payload = overflow_calls[0].args[1] + assert payload["workflow_mode"] == "compiler" + assert payload["fatal"] is False + assert payload["effective_model"] == "fallback-model:2" + assert payload["effective_provider"] == "lm_studio" + assert payload["overflow_origin"] == "provider" + assert "provider rejected" in payload["message"].lower() + assert not any(call.args[0] == "proof_attempt_failed" for call in broadcast.await_args_list) + + +@pytest.mark.asyncio +async def test_local_preflight_overflow_reports_token_budget(monkeypatch): + feedback = ProofAttemptFeedback( + attempt=1, + theorem_id="candidate-1", + error_output="MANDATORY FULL SOURCE CONTEXT OVERFLOW", + overflow_origin="local_preflight", + prompt_tokens=12_345, + max_input_tokens=10_000, + ) + + async def fake_prove(self, *args, attempt_callback=None, **kwargs): + await attempt_callback(feedback) + return False, "", "", [feedback] + + monkeypatch.setattr(ProofFormalizationAgent, "prove_candidate", fake_prove) + stage = ProofVerificationStage() + monkeypatch.setattr(stage, "_prepare_candidate", AsyncMock(return_value=_candidate())) + monkeypatch.setattr(stage, "_run_smt_check", AsyncMock(return_value=None)) + broadcast = AsyncMock() + + await stage._run_lean_pipeline_for_candidate( + theorem_candidate=_candidate(), + base_event={"source_type": "paper", "source_id": "paper-1"}, + proof_label="A", + user_prompt="Prove the claim.", + source_type="paper", + source_id="paper-1", + source_content="A short source.", + source_title="Paper", + submitter_model="configured/model", + submitter_context=32_000, + submitter_max_tokens=2_000, + role_suffix="paper", + trigger="automatic", + novel_proofs_db=None, + broadcast_fn=broadcast, + ) + + overflow_call = next( + call for call in broadcast.await_args_list if call.args[0] == "proof_context_overflow" + ) + payload = overflow_call.args[1] + assert payload["overflow_origin"] == "local_preflight" + assert payload["prompt_tokens"] == 12_345 + assert payload["max_input_tokens"] == 10_000 + assert "12,345" in payload["message"] + assert "10,000" in payload["message"] + assert "before provider invocation" in payload["message"] + + +@pytest.mark.asyncio +async def test_overflow_candidate_is_deferred_while_sibling_continues(monkeypatch): + overflow_candidate = ProofCandidate(theorem_id="overflow", statement="True") + sibling_candidate = ProofCandidate(theorem_id="sibling", statement="False") + overflow_feedback = ProofAttemptFeedback( + attempt=1, + theorem_id="overflow", + error_output="MANDATORY FULL SOURCE CONTEXT OVERFLOW", + overflow_origin="local_preflight", + prompt_tokens=11_000, + max_input_tokens=10_000, + ) + sibling_feedback = ProofAttemptFeedback( + attempt=1, + theorem_id="sibling", + error_output="ordinary Lean failure", + ) + calls = [] + + async def fake_resolve(**kwargs): + return [overflow_candidate, sibling_candidate] + + async def fake_pipeline(**kwargs): + candidate = kwargs["theorem_candidate"] + calls.append(candidate.theorem_id) + feedback = overflow_feedback if candidate.theorem_id == "overflow" else sibling_feedback + await kwargs["attempt_checkpoint_callback"](candidate, [feedback]) + return _LeanVerificationOutcome( + candidate=candidate, + proof_label=kwargs["proof_label"], + success=False, + theorem_name="", + lean_code="", + attempts=[feedback], + ) + + old_enabled = formalization_module.system_config.lean4_enabled + formalization_module.system_config.lean4_enabled = True + stage = ProofVerificationStage() + monkeypatch.setattr(stage, "_resolve_candidates", fake_resolve) + monkeypatch.setattr(stage, "_run_lean_pipeline_for_candidate", fake_pipeline) + checkpoints = [] + + async def save_checkpoint(payload): + checkpoints.append(payload) + + try: + result = await stage.run( + content="Source", + source_type="paper", + source_id="paper-1", + user_prompt="Prompt", + submitter_model="model", + submitter_context=20_000, + submitter_max_tokens=2_000, + validator_model="validator", + validator_context=20_000, + validator_max_tokens=2_000, + broadcast_fn=AsyncMock(), + novel_proofs_db=AsyncMock(), + checkpoint_callback=save_checkpoint, + ) + finally: + formalization_module.system_config.lean4_enabled = old_enabled + + assert set(calls) == {"overflow", "sibling"} + assert result.had_error is False + assert result.deferred_candidate_ids == ["overflow"] + assert [item.theorem_id for item in result.results] == ["sibling"] + assert checkpoints[-1]["status"] == "deferred" + assert checkpoints[-1]["processed_candidate_ids"] == ["sibling"] + + +@pytest.mark.asyncio +async def test_compiler_context_overflow_preserves_feedback_route(): + coordinator = object.__new__(CompilerCoordinator) + coordinator.autonomous_mode = False + coordinator.current_mode = "rigor" + coordinator.is_running = True + coordinator._broadcast = AsyncMock() + + feedback = ProofAttemptFeedback( + attempt=1, + theorem_id="candidate-1", + error_output="MANDATORY FULL SOURCE CONTEXT OVERFLOW", + configured_model="configured/model", + configured_provider="openrouter", + effective_model="fallback-model:2", + effective_provider="lm_studio", + ) + error = ProofFormalizationContextOverflowError(feedback) + + await coordinator._handle_context_overflow(error, role_id="compiler_rigor") + + event_name, payload = coordinator._broadcast.await_args.args + assert event_name == "context_overflow_error" + assert payload["configured_model"] == "configured/model" + assert payload["configured_provider"] == "openrouter" + assert payload["effective_model"] == "fallback-model:2" + assert payload["effective_provider"] == "lm_studio" diff --git a/tests/test_proof_context_regressions.py b/tests/regressions/test_proof_context_regressions.py similarity index 70% rename from tests/test_proof_context_regressions.py rename to tests/regressions/test_proof_context_regressions.py index c866044..9b479e6 100644 --- a/tests/test_proof_context_regressions.py +++ b/tests/regressions/test_proof_context_regressions.py @@ -1,4 +1,5 @@ import json +import importlib import tempfile import unittest from unittest import mock @@ -12,6 +13,7 @@ from backend.autonomous.agents import proof_identification_agent as proof_identification_module from backend.autonomous.agents import reference_selector as reference_selector_module from backend.autonomous.agents.final_answer import certainty_assessor as certainty_assessor_module +from backend.autonomous.core import proof_novelty as proof_novelty_module from backend.autonomous.core import proof_registration as proof_registration_module from backend.autonomous.core.proof_verification_stage import ProofVerificationStage from backend.autonomous.memory.brainstorm_memory import BrainstormMemory @@ -221,7 +223,7 @@ async def test_manual_compiler_current_runtime_uses_rigor_and_proofs_submitter(s self.assertTrue(snapshot.paper.supercharge_enabled) self.assertEqual(snapshot.validator.model_id, "validator-model") - async def test_cross_source_exact_proof_match_is_stored_non_novel_without_novelty_call(self): + async def test_cross_source_exact_match_runs_novelty_and_stores_current_occurrence(self): existing_record = ProofRecord( proof_id="proof_existing", theorem_statement="Exact theorem sentinel.", @@ -240,18 +242,24 @@ async def get_all_proofs(self): return list(self.records) def get_novel_proofs_for_injection(self): - raise AssertionError("Novelty context should not be requested for exact cross-source duplicates.") + return "Existing private-history proof context." - async def add_proof_if_absent(self, record): - stored = record.model_copy(update={"proof_id": "proof_reused"}) + async def add_proof_occurrence(self, record): + stored = record.model_copy(update={"proof_id": "proof_current_occurrence"}) self.records.append(stored) - return stored, False + return stored + + async def get_or_create_active_run_id(self): + return "manual-stable-run" - async def fail_novelty_call(**_kwargs): - raise AssertionError("Novelty API should not run for exact cross-source duplicates.") + novelty_calls = [] + + async def assess_exact_match(**kwargs): + novelty_calls.append(kwargs) + return "not_novel", "Validator independently classified this occurrence." old_assess = proof_registration_module.assess_proof_novelty - proof_registration_module.assess_proof_novelty = fail_novelty_call + proof_registration_module.assess_proof_novelty = assess_exact_match try: registration = await proof_registration_module.register_verified_lean_proof( proof_database=FakeProofDb(), @@ -270,9 +278,524 @@ async def fail_novelty_call(**_kwargs): proof_registration_module.assess_proof_novelty = old_assess self.assertFalse(registration.duplicate) - self.assertFalse(registration.record.novel) + self.assertEqual(len(novelty_calls), 1) + self.assertEqual(novelty_calls[0]["existing_novel_proofs"], "") + self.assertEqual(registration.record.proof_id, "proof_current_occurrence") self.assertEqual(registration.record.novelty_tier, "not_novel") - self.assertIn("proof_existing", registration.record.novelty_reasoning) + self.assertEqual(registration.record.independent_novelty_tier, "not_novel") + self.assertEqual( + registration.record.independent_novelty_reasoning, + "Validator independently classified this occurrence.", + ) + self.assertEqual(registration.record.exact_duplicate_proof_id, "proof_existing") + self.assertEqual(registration.record.exact_duplicate_run_id, "brainstorm:topic_001") + self.assertEqual(registration.record.run_id, "manual-stable-run") + self.assertEqual(registration.record.user_prompt, "Prove the prompt.") + + async def test_cross_run_exact_novel_match_gets_silent_duplicate_overlay(self): + existing_record = ProofRecord( + proof_id="proof_existing", + theorem_statement="Exact theorem\n sentinel.", + source_type="brainstorm", + source_id="topic_001", + run_id="prior-run", + lean_code="\r\ntheorem exact_theorem_sentinel : True := by trivial\r\n", + novel=True, + novelty_tier="mathematical_discovery", + ) + + class FakeProofDb: + def __init__(self): + self.records = [existing_record] + + async def get_all_proofs(self): + return list(self.records) + + def get_novel_proofs_for_injection(self): + return "" + + async def add_proof_occurrence(self, record): + stored = record.model_copy(update={"proof_id": "proof_current"}) + self.records.append(stored) + return stored + + async def assess_exact_match(**_kwargs): + return "novel_variant", "Independent novelty judgment." + + old_assess = proof_registration_module.assess_proof_novelty + proof_registration_module.assess_proof_novelty = assess_exact_match + try: + registration = await proof_registration_module.register_verified_lean_proof( + proof_database=FakeProofDb(), + user_prompt="Prove the prompt.", + theorem_statement=" Exact theorem sentinel. ", + lean_code="theorem exact_theorem_sentinel : True := by trivial\n", + validator_model="validator", + validator_context=4000, + validator_max_tokens=1000, + task_id="proof_novelty_test", + role_id="autonomous_proof_novelty", + source_type="paper", + source_id="paper_001", + run_id="current-run", + ) + finally: + proof_registration_module.assess_proof_novelty = old_assess + + self.assertFalse(registration.duplicate) + self.assertEqual(registration.record.proof_id, "proof_current") + self.assertEqual(registration.record.novelty_tier, "duplicate_novel") + self.assertEqual(registration.record.independent_novelty_tier, "novel_variant") + self.assertEqual( + registration.record.independent_novelty_reasoning, + "Independent novelty judgment.", + ) + self.assertEqual(registration.record.exact_duplicate_proof_id, "proof_existing") + self.assertEqual(registration.record.exact_duplicate_run_id, "prior-run") + self.assertEqual(registration.record.artifact_purpose, "verified_occurrence") + self.assertTrue(registration.record.canonical_identity_version) + self.assertTrue(registration.record.canonical_theorem_statement_hash) + self.assertTrue(registration.record.canonical_lean_code_hash) + + async def test_archived_cross_mode_exact_match_overlays_duplicate(self): + from backend.shared.proof_search.models import UnifiedProofSearchRecord + + archived = UnifiedProofSearchRecord( + search_id="manual:archive:proof_009", + corpus="manual", + corpus_scope="archived", + source_kind="verified_proof", + proof_id="proof_009", + session_id="manual-archive", + run_id="manual-prior-run", + source_type="paper", + source_id="manual-paper", + theorem_statement="Archived exact theorem.", + lean_code="theorem archived_exact : True := by trivial", + novelty_tier="novel_variant", + canonical_uri="moto-proof://manual/manual-archive/proof_009", + ) + + class FakeProofDb: + async def get_all_proofs(self): + return [] + + async def add_proof_occurrence(self, record): + return record.model_copy(update={"proof_id": "proof_current"}) + + class FakeSearchService: + async def exact_identity_neighborhood(self, **kwargs): + self.kwargs = kwargs + return [archived] + + async def assess_exact_match(**kwargs): + self.assertEqual(kwargs["existing_novel_proofs"], "") + return "mathematical_discovery", "Independent archived-match judgment." + + from backend.shared.proof_search import search_service as search_service_module + + old_assess = proof_registration_module.assess_proof_novelty + old_service = search_service_module.proof_search_service + proof_registration_module.assess_proof_novelty = assess_exact_match + fake_service = FakeSearchService() + search_service_module.proof_search_service = fake_service + try: + registration = await proof_registration_module.register_verified_lean_proof( + proof_database=FakeProofDb(), + user_prompt="Prove the prompt.", + theorem_statement="Archived exact theorem.", + lean_code="\r\ntheorem archived_exact : True := by trivial\r\n", + validator_model="validator", + validator_context=4000, + validator_max_tokens=1000, + task_id="proof_novelty_test", + role_id="autonomous_proof_novelty", + source_type="leanoj_final", + source_id="leanoj-current", + run_id="leanoj-current-run", + ) + finally: + proof_registration_module.assess_proof_novelty = old_assess + search_service_module.proof_search_service = old_service + + self.assertEqual(registration.record.novelty_tier, "duplicate_novel") + self.assertEqual( + registration.record.independent_novelty_tier, + "mathematical_discovery", + ) + self.assertEqual(registration.record.exact_duplicate_proof_id, "proof_009") + self.assertEqual(registration.record.exact_duplicate_run_id, "manual-prior-run") + self.assertEqual(set(fake_service.kwargs["corpora"]), {"moto", "manual", "leanoj"}) + self.assertEqual( + fake_service.kwargs["exclude_run_ids"], + ["leanoj-current-run"], + ) + + async def test_duplicate_novel_registration_uses_known_style_broadcast_event(self): + events = [] + record = ProofRecord( + proof_id="proof_duplicate_novel", + theorem_statement="Duplicate novel theorem sentinel.", + source_type="paper", + source_id="paper_001", + lean_code="theorem duplicate_novel_event_sentinel : True := by trivial", + novel=True, + novelty_tier="duplicate_novel", + ) + + async def broadcast(event_type, payload): + events.append((event_type, payload)) + + await proof_registration_module._broadcast_registered_proof( + broadcast_fn=broadcast, + record=record, + base_event={"source_type": "paper"}, + ) + + self.assertEqual([event_type for event_type, _payload in events], ["known_proof_verified"]) + self.assertTrue(events[0][1]["is_novel"]) + self.assertEqual(events[0][1]["novelty_tier"], "duplicate_novel") + + async def test_proof_library_novel_category_uses_prompt_injection_tiers(self): + with tempfile.TemporaryDirectory() as tmpdir: + db = ProofDatabase() + db.set_base_dir(Path(tmpdir)) + await db.initialize() + + records = [ + ProofRecord( + proof_id="", + theorem_statement="Legacy novel flag but not novel tier.", + source_type="brainstorm", + source_id="topic_legacy", + lean_code="theorem legacy_true_not_novel : True := by trivial", + novel=True, + novelty_tier="not_novel", + ), + ProofRecord( + proof_id="", + theorem_statement="Unknown legacy novel tier.", + source_type="brainstorm", + source_id="topic_unknown", + lean_code="theorem legacy_true_unknown : True := by trivial", + novel=True, + novelty_tier="novel", + ), + ProofRecord( + proof_id="", + theorem_statement="Strict novel theorem.", + source_type="brainstorm", + source_id="topic_novel", + lean_code="theorem strict_novel_variant : True := by trivial", + novel=True, + novelty_tier="novel_variant", + ), + ] + for record in records: + await db.add_proof(record) + + novel_entries = await db._list_proofs_from_directory(Path(tmpdir), "test_session", "novel") + all_entries = await db._list_proofs_from_directory(Path(tmpdir), "test_session", "all") + + self.assertEqual(len(all_entries), 3) + self.assertEqual([entry["theorem_statement"] for entry in novel_entries], ["Strict novel theorem."]) + + async def test_cross_source_near_duplicate_still_runs_true_novelty_assessment(self): + existing_record = ProofRecord( + proof_id="proof_existing", + theorem_statement="Near duplicate theorem sentinel.", + source_type="brainstorm", + source_id="topic_001", + lean_code="theorem near_duplicate_sentinel : True := by trivial", + novel=True, + novelty_tier="mathematical_discovery", + ) + + class FakeProofDb: + def __init__(self): + self.records = [existing_record] + + async def get_all_proofs(self): + return list(self.records) + + def get_novel_proofs_for_injection(self): + return "existing novel context" + + async def add_proof_if_absent(self, record): + stored = record.model_copy(update={"proof_id": "proof_near_duplicate"}) + self.records.append(stored) + return stored, False + + novelty_calls = [] + + async def assess_near_duplicate(**kwargs): + novelty_calls.append(kwargs) + return "novel_variant", "Near duplicate advances the user's problem-solving context." + + old_assess = proof_registration_module.assess_proof_novelty + proof_registration_module.assess_proof_novelty = assess_near_duplicate + try: + registration = await proof_registration_module.register_verified_lean_proof( + proof_database=FakeProofDb(), + user_prompt="Prove the prompt.", + theorem_statement="Near duplicate theorem sentinel with stronger context.", + lean_code="theorem near_duplicate_sentinel_stronger : True := by trivial", + validator_model="validator", + validator_context=4000, + validator_max_tokens=1000, + task_id="proof_novelty_test", + role_id="autonomous_proof_novelty", + source_type="paper", + source_id="paper_001", + ) + finally: + proof_registration_module.assess_proof_novelty = old_assess + + self.assertEqual(len(novelty_calls), 1) + self.assertTrue(registration.record.novel) + self.assertEqual(registration.record.novelty_tier, "novel_variant") + self.assertIn("advances", registration.record.novelty_reasoning) + + async def test_proof_novelty_json_parse_failure_retries_before_rating(self): + class FakeApiClientManager: + def __init__(self): + self.calls = [] + + async def generate_completion(self, **kwargs): + self.calls.append(kwargs) + if len(self.calls) == 1: + return {"choices": [{"message": {"content": '{"novelty_tier":'}}]} + return { + "choices": [ + { + "message": { + "content": json.dumps( + { + "novelty_tier": "mathematical_discovery", + "reasoning": "Retry recovered a valid novelty judgement.", + } + ) + } + } + ] + } + + fake_api_client_manager = FakeApiClientManager() + old_api_client_manager = proof_novelty_module.api_client_manager + proof_novelty_module.api_client_manager = fake_api_client_manager + try: + novelty_tier, reasoning = await proof_novelty_module.assess_proof_novelty( + user_prompt="Solve the prompt.", + theorem_statement="A verified theorem.", + lean_code="theorem retry_novelty_success : True := by trivial", + validator_model="validator", + validator_context=8000, + validator_max_tokens=1000, + existing_novel_proofs="", + task_id="proof_novelty_test", + ) + finally: + proof_novelty_module.api_client_manager = old_api_client_manager + + self.assertEqual(novelty_tier, "mathematical_discovery") + self.assertIn("Retry recovered", reasoning) + self.assertEqual(len(fake_api_client_manager.calls), 2) + self.assertEqual(fake_api_client_manager.calls[1]["task_id"], "proof_novelty_test_retry") + self.assertEqual(fake_api_client_manager.calls[1]["temperature"], 0.0) + self.assertEqual(fake_api_client_manager.calls[1]["role_id"], "autonomous_proof_novelty") + self.assertIn("previous proof-novelty response", fake_api_client_manager.calls[1]["messages"][-1]["content"]) + + async def test_proof_novelty_invalid_tier_retries_before_downgrade(self): + class FakeApiClientManager: + def __init__(self): + self.calls = [] + + async def generate_completion(self, **kwargs): + self.calls.append(kwargs) + if len(self.calls) == 1: + return { + "choices": [ + { + "message": { + "content": json.dumps( + { + "novelty_tier": "brand_new", + "reasoning": "Bad tier should not be accepted.", + } + ) + } + } + ] + } + return { + "choices": [ + { + "message": { + "content": json.dumps( + { + "novelty_tier": "novel_variant", + "reasoning": "Retry selected a valid tier.", + } + ) + } + } + ] + } + + fake_api_client_manager = FakeApiClientManager() + old_api_client_manager = proof_novelty_module.api_client_manager + proof_novelty_module.api_client_manager = fake_api_client_manager + try: + novelty_tier, reasoning = await proof_novelty_module.assess_proof_novelty( + user_prompt="Solve the prompt.", + theorem_statement="A verified theorem.", + lean_code="theorem retry_invalid_tier : True := by trivial", + validator_model="validator", + validator_context=8000, + validator_max_tokens=1000, + existing_novel_proofs="", + task_id="proof_novelty_bad_tier", + ) + finally: + proof_novelty_module.api_client_manager = old_api_client_manager + + self.assertEqual(novelty_tier, "novel_variant") + self.assertIn("valid tier", reasoning) + self.assertEqual(len(fake_api_client_manager.calls), 2) + + async def test_proof_novelty_model_cannot_assign_duplicate_overlay(self): + class FakeApiClientManager: + def __init__(self): + self.calls = [] + + async def generate_completion(self, **kwargs): + self.calls.append(kwargs) + tier = "duplicate_novel" if len(self.calls) == 1 else "novel_variant" + return { + "choices": [ + { + "message": { + "content": json.dumps( + { + "novelty_tier": tier, + "reasoning": "Independent novelty only.", + } + ) + } + } + ] + } + + fake_api_client_manager = FakeApiClientManager() + old_api_client_manager = proof_novelty_module.api_client_manager + proof_novelty_module.api_client_manager = fake_api_client_manager + try: + novelty_tier, _ = await proof_novelty_module.assess_proof_novelty( + user_prompt="Solve the prompt.", + theorem_statement="A verified theorem.", + lean_code="theorem reject_model_duplicate_overlay : True := by trivial", + validator_model="validator", + validator_context=8000, + validator_max_tokens=1000, + existing_novel_proofs="", + task_id="proof_novelty_duplicate_overlay", + ) + finally: + proof_novelty_module.api_client_manager = old_api_client_manager + + self.assertEqual(novelty_tier, "novel_variant") + self.assertEqual(len(fake_api_client_manager.calls), 2) + self.assertNotIn( + '"not_novel | duplicate_novel', + fake_api_client_manager.calls[1]["messages"][-1]["content"], + ) + + async def test_proof_novelty_missing_tier_retries_before_downgrade(self): + class FakeApiClientManager: + def __init__(self): + self.calls = [] + + async def generate_completion(self, **kwargs): + self.calls.append(kwargs) + if len(self.calls) == 1: + return { + "choices": [ + { + "message": { + "content": json.dumps( + { + "reasoning": "Missing tier should not be accepted.", + } + ) + } + } + ] + } + return { + "choices": [ + { + "message": { + "content": json.dumps( + { + "novelty_tier": "novel_formulation", + "reasoning": "Retry supplied the missing tier.", + } + ) + } + } + ] + } + + fake_api_client_manager = FakeApiClientManager() + old_api_client_manager = proof_novelty_module.api_client_manager + proof_novelty_module.api_client_manager = fake_api_client_manager + try: + novelty_tier, reasoning = await proof_novelty_module.assess_proof_novelty( + user_prompt="Solve the prompt.", + theorem_statement="A verified theorem.", + lean_code="theorem retry_missing_tier : True := by trivial", + validator_model="validator", + validator_context=8000, + validator_max_tokens=1000, + existing_novel_proofs="", + task_id="proof_novelty_missing_tier", + ) + finally: + proof_novelty_module.api_client_manager = old_api_client_manager + + self.assertEqual(novelty_tier, "novel_formulation") + self.assertIn("missing tier", reasoning) + self.assertEqual(len(fake_api_client_manager.calls), 2) + + async def test_proof_novelty_retry_exhaustion_downgrades_to_not_novel(self): + class FakeApiClientManager: + def __init__(self): + self.calls = [] + + async def generate_completion(self, **kwargs): + self.calls.append(kwargs) + return {"choices": [{"message": {"content": '{"novelty_tier":'}}]} + + fake_api_client_manager = FakeApiClientManager() + old_api_client_manager = proof_novelty_module.api_client_manager + proof_novelty_module.api_client_manager = fake_api_client_manager + try: + novelty_tier, reasoning = await proof_novelty_module.assess_proof_novelty( + user_prompt="Solve the prompt.", + theorem_statement="A verified theorem.", + lean_code="theorem retry_novelty_exhausted : True := by trivial", + validator_model="validator", + validator_context=8000, + validator_max_tokens=1000, + existing_novel_proofs="", + task_id="proof_novelty_exhausted", + ) + finally: + proof_novelty_module.api_client_manager = old_api_client_manager + + self.assertEqual(novelty_tier, "not_novel") + self.assertIn("retry failed after retry exhaustion", reasoning) + self.assertEqual(len(fake_api_client_manager.calls), 2) def test_formalization_prompt_separates_verified_proof_context(self): injected_prompt = ( @@ -1153,6 +1676,71 @@ async def generate_completion(self, **_kwargs): self.assertTrue(has_candidates) self.assertEqual([candidate.theorem_id for candidate in candidates], ["valid"]) + async def test_proof_identification_retries_codex_max_output_truncation(self): + class FakeApiClientManager: + def __init__(self): + self.calls = [] + + async def prewarm_assistant_memory_context(self, **_kwargs): + return "" + + async def generate_completion(self, **kwargs): + self.calls.append(kwargs) + if len(self.calls) == 1: + raise RuntimeError( + "OpenAI Codex completion failed: " + '{"type":"response.incomplete","response":{"status":"incomplete",' + '"incomplete_details":{"reason":"max_output_tokens"}}}' + ) + return { + "choices": [ + { + "message": { + "content": json.dumps( + { + "has_provable_theorems": True, + "theorems": [ + { + "theorem_id": "retry_valid", + "statement": "A retry-discovered theorem.", + "formal_sketch": "Sketch.", + "expected_novelty_tier": "novel_variant", + "prompt_relevance_rationale": "It directly advances the prompt.", + "novelty_rationale": "It is not routine.", + "why_not_standard_known_result": "It is not a standard reference result.", + } + ], + } + ) + } + } + ] + } + + fake_api_client_manager = FakeApiClientManager() + old_api_client_manager = proof_identification_module.api_client_manager + proof_identification_module.api_client_manager = fake_api_client_manager + try: + agent = proof_identification_module.ProofIdentificationAgent( + model_id="gpt-5.5", + context_window=8000, + max_output_tokens=1000, + role_id="autonomous_proof_identification_test", + ) + has_candidates, candidates = await agent.identify_candidates( + user_research_prompt="Solve the prompt.", + source_type="brainstorm", + source_id="manual_aggregator", + source_content="Source content.", + ) + finally: + proof_identification_module.api_client_manager = old_api_client_manager + + self.assertTrue(has_candidates) + self.assertEqual([candidate.theorem_id for candidate in candidates], ["retry_valid"]) + self.assertEqual(len(fake_api_client_manager.calls), 2) + self.assertEqual(fake_api_client_manager.calls[1]["task_id"], "proof_id_000_retry") + def test_paper_proof_stripping_preserves_self_review_after_fallback_proofs(self): stripped = PaperLibrary.strip_verified_proofs_from_content( "Manual compiler paper body sentinel.\n\n" @@ -1305,16 +1893,18 @@ async def record_failed_candidate(self, *_args, **_kwargs): finally: system_config.lean4_enabled = old_lean4_enabled - self.assertTrue(result.had_error) - self.assertIn("MANDATORY FULL SOURCE CONTEXT OVERFLOW", result.error_message) + self.assertFalse(result.had_error) + self.assertEqual(result.deferred_candidate_ids, ["overflow"]) self.assertEqual(record_failed_calls, 0) + self.assertIn("proof_context_overflow", events) self.assertNotIn("proof_attempts_exhausted", events) - async def test_codex_max_output_incomplete_does_not_crash_proof_stage(self): + async def test_codex_max_output_incomplete_counts_as_failed_attempt(self): old_lean4_enabled = system_config.lean4_enabled system_config.lean4_enabled = True stage = ProofVerificationStage() events: list[str] = [] + attempt_failure_summaries: list[str] = [] checkpoint_statuses: list[str] = [] candidate = ProofCandidate( theorem_id="codex_incomplete", @@ -1322,8 +1912,10 @@ async def test_codex_max_output_incomplete_does_not_crash_proof_stage(self): expected_novelty_tier="mathematical_discovery", ) - async def broadcast(event_type, _payload): + async def broadcast(event_type, payload): events.append(event_type) + if event_type == "proof_attempt_failed": + attempt_failure_summaries.append(payload.get("error_summary", "")) async def checkpoint(payload): checkpoint_statuses.append(payload["status"]) @@ -1371,12 +1963,84 @@ class FakeProofDb: proof_formalization_module.api_client_manager.generate_completion = old_generate_completion system_config.lean4_enabled = old_lean4_enabled - self.assertTrue(result.had_error) - self.assertIn("MODEL OUTPUT INCOMPLETE", result.error_message) - self.assertNotIn("response.incomplete", result.error_message) - self.assertIn("error", checkpoint_statuses) + self.assertFalse(result.had_error) + self.assertEqual(result.verified_count, 0) + self.assertEqual(result.total_candidates, 1) + self.assertIn("running", checkpoint_statuses) + self.assertNotIn("error", checkpoint_statuses) + self.assertEqual(events.count("proof_attempt_failed"), 5) + self.assertTrue(any("MODEL OUTPUT TRUNCATED" in summary for summary in attempt_failure_summaries)) + self.assertIn("proof_check_complete", events) + self.assertIn("proof_attempts_exhausted", events) + + async def test_chat_finish_reason_length_counts_as_failed_attempt(self): + old_lean4_enabled = system_config.lean4_enabled + system_config.lean4_enabled = True + stage = ProofVerificationStage() + events: list[str] = [] + attempt_failure_summaries: list[str] = [] + candidate = ProofCandidate( + theorem_id="chat_length_stop", + statement="Chat length stop theorem.", + expected_novelty_tier="mathematical_discovery", + ) + + async def broadcast(event_type, payload): + events.append(event_type) + if event_type == "proof_attempt_failed": + attempt_failure_summaries.append(payload.get("error_summary", "")) + + async def fake_prepare_candidate(**kwargs): + return kwargs["theorem_candidate"] + + async def fake_smt_check(**_kwargs): + return None + + async def return_length_stopped_response(*_args, **_kwargs): + return { + "choices": [ + { + "index": 0, + "message": {"role": "assistant", "content": "{\"lean_code\": \""}, + "finish_reason": "length", + } + ], + "usage": {"prompt_tokens": 10, "completion_tokens": 1000, "total_tokens": 1010}, + } + + class FakeProofDb: + pass + + old_generate_completion = proof_formalization_module.api_client_manager.generate_completion + proof_formalization_module.api_client_manager.generate_completion = return_length_stopped_response + stage._prepare_candidate = fake_prepare_candidate + stage._run_smt_check = fake_smt_check + try: + result = await stage.run( + content="paper source", + source_type="paper", + source_id="paper_001", + user_prompt="prove things", + submitter_model="model", + submitter_context=4000, + submitter_max_tokens=1000, + validator_model="validator", + validator_context=4000, + validator_max_tokens=1000, + broadcast_fn=broadcast, + novel_proofs_db=FakeProofDb(), + theorem_candidates=[candidate], + append_to_source=False, + ) + finally: + proof_formalization_module.api_client_manager.generate_completion = old_generate_completion + system_config.lean4_enabled = old_lean4_enabled + + self.assertFalse(result.had_error) + self.assertEqual(events.count("proof_attempt_failed"), 5) + self.assertTrue(any("MODEL OUTPUT TRUNCATED" in summary for summary in attempt_failure_summaries)) + self.assertIn("proof_attempts_exhausted", events) self.assertIn("proof_check_complete", events) - self.assertNotIn("proof_attempts_exhausted", events) async def test_codex_transient_gateway_disconnect_preserves_proof_checkpoint(self): old_lean4_enabled = system_config.lean4_enabled @@ -1845,5 +2509,115 @@ class FakeProofDb: self.assertIn("proof_check_no_candidates", events) +class AutonomousProofFailedHintCleanupTests(unittest.IsolatedAsyncioTestCase): + async def test_clear_all_data_removes_failed_hints_but_keeps_verified_proof_history(self): + coordinator_module = importlib.import_module("backend.autonomous.core.autonomous_coordinator") + queue_manager_module = importlib.import_module("backend.aggregator.core.queue_manager") + from backend.autonomous.core.autonomous_coordinator import AutonomousCoordinator + + with tempfile.TemporaryDirectory() as tmp: + data_root = Path(tmp) + sessions_dir = data_root / "auto_sessions" + session_dir = sessions_dir / "cleanup_session" + proofs_dir = session_dir / "proofs" + failed_dir = proofs_dir / "failed" + failed_dir.mkdir(parents=True) + (session_dir / "session_metadata.json").write_text( + json.dumps({"session_id": "cleanup_session", "user_prompt": "Prompt"}), + encoding="utf-8", + ) + (session_dir / "session_stats.json").write_text( + json.dumps({"current_brainstorm_id": "topic_1", "current_paper_id": "paper_1"}), + encoding="utf-8", + ) + (session_dir / "workflow_state.json").write_text( + json.dumps({"current_tier": "tier1_aggregation"}), + encoding="utf-8", + ) + (proofs_dir / "proofs_index.json").write_text( + json.dumps({"next_proof_id": 2, "proofs": [{"proof_id": "proof_001"}]}), + encoding="utf-8", + ) + (proofs_dir / "proof_001.json").write_text( + json.dumps({"proof_id": "proof_001", "theorem_statement": "Verified"}), + encoding="utf-8", + ) + (proofs_dir / "proof_001_lean.lean").write_text( + "theorem verified_cleanup : True := by trivial", + encoding="utf-8", + ) + (failed_dir / "topic_1.json").write_text( + json.dumps({"items": [{"theorem_id": "failed_1"}]}), + encoding="utf-8", + ) + + old_values = { + "auto_sessions_base_dir": system_config.auto_sessions_base_dir, + "auto_brainstorms_dir": system_config.auto_brainstorms_dir, + "auto_papers_dir": system_config.auto_papers_dir, + "auto_research_topic_rejections_file": system_config.auto_research_topic_rejections_file, + "data_dir": system_config.data_dir, + } + try: + system_config.data_dir = str(data_root) + system_config.auto_sessions_base_dir = str(sessions_dir) + system_config.auto_brainstorms_dir = str(data_root / "auto_brainstorms") + system_config.auto_papers_dir = str(data_root / "auto_papers") + system_config.auto_research_topic_rejections_file = str( + data_root / "auto_research_topic_rejections.txt" + ) + + coordinator = AutonomousCoordinator() + with ( + mock.patch.object(coordinator_module.session_manager, "clear", new=mock.AsyncMock()), + mock.patch.object(coordinator_module.brainstorm_memory, "set_session_manager"), + mock.patch.object(coordinator_module.paper_library, "set_session_manager"), + mock.patch.object(coordinator_module.research_metadata, "set_session_manager"), + mock.patch.object(coordinator_module.final_answer_memory, "set_session_manager"), + mock.patch.object(coordinator_module.proof_database, "set_session_manager"), + mock.patch.object( + coordinator_module.proof_database, + "clear_failed_candidates", + new=mock.AsyncMock(), + ) as clear_active_failed, + mock.patch.object( + coordinator_module.research_metadata, + "clear_all", + new=mock.AsyncMock(), + ), + mock.patch.object( + coordinator_module.autonomous_rejection_logs, + "clear_all", + new=mock.AsyncMock(), + ), + mock.patch.object( + coordinator_module.autonomous_api_logger, + "clear_logs", + new=mock.AsyncMock(), + ), + mock.patch.object(coordinator_module.autonomous_rag_manager, "reset"), + mock.patch.object( + coordinator_module.rag_manager, + "clear_all_documents_async", + new=mock.AsyncMock(), + ), + mock.patch.object(queue_manager_module.queue_manager, "clear", new=mock.AsyncMock()), + ): + await coordinator.clear_all_data() + + metadata = json.loads((session_dir / "session_metadata.json").read_text(encoding="utf-8")) + self.assertEqual(metadata["status"], "cleared") + self.assertTrue(metadata["resume_disabled"]) + self.assertFalse((session_dir / "workflow_state.json").exists()) + self.assertFalse(failed_dir.exists()) + self.assertTrue((proofs_dir / "proofs_index.json").exists()) + self.assertTrue((proofs_dir / "proof_001.json").exists()) + self.assertTrue((proofs_dir / "proof_001_lean.lean").exists()) + clear_active_failed.assert_awaited_once() + finally: + for name, value in old_values.items(): + setattr(system_config, name, value) + + if __name__ == "__main__": unittest.main() diff --git a/tests/regressions/test_proofs_warm_start_definition.py b/tests/regressions/test_proofs_warm_start_definition.py new file mode 100644 index 0000000..18953e7 --- /dev/null +++ b/tests/regressions/test_proofs_warm_start_definition.py @@ -0,0 +1,14 @@ +import ast +from pathlib import Path + + +def test_lean4_warm_start_helper_has_single_definition(): + route_path = Path(__file__).resolve().parents[2] / "backend" / "api" / "routes" / "proofs.py" + module = ast.parse(route_path.read_text(encoding="utf-8")) + definitions = [ + node + for node in module.body + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) + and node.name == "_schedule_lean4_warm_start" + ] + assert len(definitions) == 1 diff --git a/tests/regressions/test_provider_routing_errors.py b/tests/regressions/test_provider_routing_errors.py new file mode 100644 index 0000000..b65d1dd --- /dev/null +++ b/tests/regressions/test_provider_routing_errors.py @@ -0,0 +1,300 @@ +import unittest +from unittest.mock import AsyncMock, patch + +import httpx + +import backend.shared.api_client_manager as api_manager_module +from backend.shared.api_client_manager import APIClientManager, _typed_provider_context_error +from backend.shared.boost_manager import boost_manager +from backend.shared.lm_studio_client import LMStudioClient +from backend.shared.model_error_utils import ( + is_non_retryable_model_error, + is_provider_context_length_error, + is_transient_model_call_error, +) +from backend.shared.models import BoostConfig, ModelConfig +from backend.shared.openrouter_client import OpenRouterClient +from backend.shared.provider_errors import ( + ProviderContextLengthError, + ProviderRouteError, + ProviderRouteIdentity, +) + + +class ProviderErrorUtilityTests(unittest.TestCase): + def test_context_error_is_typed_and_never_exposes_secret(self): + error = ProviderContextLengthError( + "context_length_exceeded Bearer secret-value", + route=ProviderRouteIdentity( + provider="openrouter", + model="vendor/model", + host_provider="host", + ), + ) + + self.assertTrue(is_provider_context_length_error(error)) + self.assertTrue(is_non_retryable_model_error(error)) + self.assertNotIn("secret-value", str(error)) + self.assertIn("vendor/model", str(error)) + + def test_wrapped_transient_error_uses_original_cause(self): + cause = ValueError("OpenRouter connection failed after retries") + error = ProviderRouteError( + "Provider request failed", + route=ProviderRouteIdentity(provider="openrouter", model="vendor/model"), + cause=cause, + ) + + self.assertTrue(is_transient_model_call_error(error)) + self.assertFalse(is_non_retryable_model_error(error)) + + def test_timeout_subclass_is_transient(self): + request = httpx.Request("POST", "https://provider.invalid") + cause = httpx.ReadTimeout("slow", request=request) + error = ProviderRouteError( + "Provider timed out", + route=ProviderRouteIdentity(provider="openrouter", model="vendor/model"), + cause=cause, + ) + + self.assertTrue(is_transient_model_call_error(error)) + + def test_oauth_and_sakana_context_wrapping_is_typed_and_redacted(self): + for provider, model in ( + ("sakana_fugu", "fugu-model"), + ("openai_codex_oauth", "codex-model"), + ("xai_grok_oauth", "grok-model"), + ): + with self.subTest(provider=provider): + error = _typed_provider_context_error( + ValueError("context_length_exceeded Bearer secret-value"), + provider=provider, + model=model, + ) + self.assertEqual(error.route.provider, provider) + self.assertEqual(error.route.model, model) + self.assertNotIn("secret-value", str(error)) + + +class ProviderClientTypedErrorTests(unittest.IsolatedAsyncioTestCase): + async def test_openrouter_context_rejection_is_typed(self): + client = OpenRouterClient("test-key") + request = httpx.Request("POST", "https://openrouter.ai/api/v1/chat/completions") + response = httpx.Response( + 400, + request=request, + json={"error": {"message": "context_length_exceeded"}}, + ) + client.client.post = AsyncMock(return_value=response) + client.MAX_RETRIES = 1 + try: + with self.assertRaises(ProviderContextLengthError) as raised: + await client.generate_completion( + model="vendor/model", + messages=[{"role": "user", "content": "hello"}], + max_tokens=10, + provider="safe-host", + ) + self.assertEqual(raised.exception.route.provider, "openrouter") + self.assertEqual(raised.exception.route.host_provider, "safe-host") + finally: + await client.close() + + async def test_openrouter_timeout_is_typed(self): + client = OpenRouterClient("test-key") + request = httpx.Request("POST", "https://openrouter.ai/api/v1/chat/completions") + client.client.post = AsyncMock(side_effect=httpx.ReadTimeout("slow", request=request)) + client.MAX_RETRIES = 1 + try: + with self.assertRaises(ProviderRouteError) as raised: + await client.generate_completion( + model="vendor/model", + messages=[{"role": "user", "content": "hello"}], + max_tokens=10, + ) + self.assertIsInstance(raised.exception.cause, httpx.TimeoutException) + finally: + await client.close() + + async def test_lm_studio_connection_failure_is_typed(self): + client = LMStudioClient(base_url="http://127.0.0.1:1") + client.client.post = AsyncMock(side_effect=httpx.ConnectError("offline")) + try: + with patch("backend.shared.lm_studio_client.asyncio.sleep", new=AsyncMock()): + with self.assertRaises(ProviderRouteError) as raised: + await client.generate_completion( + model="local-model", + messages=[{"role": "user", "content": "hello"}], + max_tokens=10, + skip_semaphore=True, + ) + self.assertEqual(raised.exception.route.provider, "lm_studio") + self.assertTrue(is_transient_model_call_error(raised.exception)) + finally: + await client.client.aclose() + + async def test_lm_studio_timeout_is_typed(self): + client = LMStudioClient(base_url="http://127.0.0.1:1") + request = httpx.Request("POST", "http://127.0.0.1:1/v1/chat/completions") + client.client.post = AsyncMock(side_effect=httpx.ReadTimeout("slow", request=request)) + try: + with patch("backend.shared.lm_studio_client.asyncio.sleep", new=AsyncMock()): + with self.assertRaises(ProviderRouteError) as raised: + await client.generate_completion( + model="local-model", + messages=[{"role": "user", "content": "hello"}], + max_tokens=10, + skip_semaphore=True, + ) + self.assertIsInstance(raised.exception.cause, httpx.TimeoutException) + finally: + await client.client.aclose() + + +class APIClientManagerRouteContextTests(unittest.IsolatedAsyncioTestCase): + async def test_manager_adds_role_task_and_fallback_route_identity(self): + manager = APIClientManager() + manager.configure_role( + "writer", + ModelConfig( + provider="openrouter", + model_id="vendor/model", + openrouter_model_id="vendor/model", + lm_studio_fallback_id="local-model", + context_window=4096, + max_output_tokens=512, + ), + ) + manager._role_fallback_state["writer"] = "lm_studio" + + base_error = ProviderRouteError( + "LM Studio offline", + route=ProviderRouteIdentity(provider="lm_studio", model="local-model"), + cause=httpx.ConnectError("offline"), + ) + with patch( + "backend.shared.api_client_manager.lm_studio_client.generate_completion", + new=AsyncMock(side_effect=base_error), + ): + with self.assertRaises(ProviderRouteError) as raised: + await manager._generate_completion_once( + task_id="comp_writer_001", + role_id="writer", + model="local-model", + messages=[{"role": "user", "content": "hello"}], + max_tokens=100, + ) + + route = raised.exception.route + self.assertEqual(route.role_id, "writer") + self.assertEqual(route.task_id, "comp_writer_001") + self.assertEqual(route.route_kind, "fallback") + + async def test_free_rotation_context_error_preserves_effective_route(self): + manager = APIClientManager() + manager._openrouter_client = AsyncMock() + context_error = ProviderContextLengthError( + "Bearer secret-value context_length_exceeded", + route=ProviderRouteIdentity(provider="openrouter", model="alternate/free"), + ) + manager._openrouter_client.generate_completion = AsyncMock(side_effect=context_error) + + with ( + patch.object(api_manager_module.free_model_manager, "looping_enabled", True), + patch.object(api_manager_module.free_model_manager, "auto_selector_enabled", False), + patch.object( + api_manager_module.free_model_manager, + "get_alternative_free_model", + side_effect=["alternate/free", None], + ), + ): + with self.assertRaises(ProviderContextLengthError) as raised: + await manager._try_free_model_rotation( + task_id="agg_sub1_001", + role_id="submitter", + original_model="original/free", + configured_model="original/free", + configured_provider="openrouter", + messages=[{"role": "user", "content": "hello"}], + temperature=0.0, + max_tokens=10, + response_format=None, + ) + + self.assertEqual(raised.exception.route.route_kind, "free_rotation") + self.assertEqual(raised.exception.route.model, "alternate/free") + self.assertNotIn("secret-value", str(raised.exception)) + + async def test_auto_selector_context_error_has_route_kind(self): + manager = APIClientManager() + manager._openrouter_client = AsyncMock() + manager._openrouter_client.generate_completion = AsyncMock( + side_effect=ProviderContextLengthError( + "context limit", + route=ProviderRouteIdentity(provider="openrouter", model="openrouter/free"), + ) + ) + + with ( + patch.object(api_manager_module.free_model_manager, "looping_enabled", False), + patch.object(api_manager_module.free_model_manager, "auto_selector_enabled", True), + ): + with self.assertRaises(ProviderContextLengthError) as raised: + await manager._try_free_model_rotation( + task_id="agg_sub1_001", + role_id="submitter", + original_model="original/free", + configured_model="original/free", + configured_provider="openrouter", + messages=[{"role": "user", "content": "hello"}], + temperature=0.0, + max_tokens=10, + response_format=None, + ) + + self.assertEqual(raised.exception.route.route_kind, "auto_selector") + self.assertEqual(raised.exception.route.model, "openrouter/free") + + async def test_strict_boost_preserves_typed_context_error(self): + manager = APIClientManager() + previous_config = boost_manager.boost_config + boost_manager.boost_config = BoostConfig( + enabled=True, + openrouter_api_key="test-key", + boost_model_id="boost/model", + boost_context_window=4096, + boost_max_output_tokens=512, + ) + boost_error = ProviderContextLengthError( + "context limit", + route=ProviderRouteIdentity(provider="openrouter", model="boost/model"), + ) + fake_client = AsyncMock() + fake_client.generate_completion = AsyncMock(side_effect=boost_error) + try: + with ( + patch.object(api_manager_module, "OpenRouterClient", return_value=fake_client), + patch.object( + api_manager_module.boost_logger, + "log_boost_call", + new=AsyncMock(), + ), + ): + with self.assertRaises(ProviderContextLengthError) as raised: + await manager._generate_completion_once( + task_id="comp_writer_001", + role_id="writer", + model="primary-model", + messages=[{"role": "user", "content": "hello"}], + max_tokens=10, + _moto_force_boost_mode="supercharge", + _moto_strict_boost=True, + ) + self.assertEqual(raised.exception.route.route_kind, "boost") + self.assertEqual(raised.exception.route.model, "boost/model") + finally: + boost_manager.boost_config = previous_config + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/regressions/test_rag_clear_bounded.py b/tests/regressions/test_rag_clear_bounded.py new file mode 100644 index 0000000..55ee032 --- /dev/null +++ b/tests/regressions/test_rag_clear_bounded.py @@ -0,0 +1,48 @@ +import pytest + +from backend.aggregator.core.rag_manager import CHROMA_DELETE_BATCH_SIZE, RAGManager + + +class _FakeCollection: + def __init__(self, count: int) -> None: + self.ids = [f"chunk-{index}" for index in range(count)] + self.get_calls = [] + + def get(self, **kwargs): + self.get_calls.append(kwargs) + limit = kwargs["limit"] + return {"ids": self.ids[:limit]} + + def delete(self, *, ids): + deleted = set(ids) + self.ids = [item for item in self.ids if item not in deleted] + + +@pytest.mark.asyncio +async def test_source_delete_fetches_ids_in_bounded_batches(monkeypatch): + manager = RAGManager() + collection = _FakeCollection(CHROMA_DELETE_BATCH_SIZE + 1) + + async def direct_native(_name, func, *args, **kwargs): + return func(*args, **kwargs) + + monkeypatch.setattr(manager, "_ensure_initialized_locked", lambda: None) + monkeypatch.setattr(manager, "_await_native_worker", direct_native) + manager.collections[256] = collection + deleted = await manager._delete_matching_ids( + 256, + "source delete", + where={"source_file": "source.txt"}, + ) + + assert deleted == CHROMA_DELETE_BATCH_SIZE + 1 + assert collection.ids == [] + assert len(collection.get_calls) == 3 + assert all( + call == { + "limit": CHROMA_DELETE_BATCH_SIZE, + "include": [], + "where": {"source_file": "source.txt"}, + } + for call in collection.get_calls + ) diff --git a/tests/regressions/test_solution_path_semantic_coverage.py b/tests/regressions/test_solution_path_semantic_coverage.py new file mode 100644 index 0000000..20a1cc3 --- /dev/null +++ b/tests/regressions/test_solution_path_semantic_coverage.py @@ -0,0 +1,117 @@ +import ast +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] + + +def _source(relative_path: str, class_name: str, method_name: str) -> str: + text = (ROOT / relative_path).read_text(encoding="utf-8") + tree = ast.parse(text) + for node in tree.body: + if isinstance(node, ast.ClassDef) and node.name == class_name: + for child in node.body: + if isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef)) and child.name == method_name: + return ast.get_source_segment(text, child) or "" + raise AssertionError(f"{class_name}.{method_name} not found in {relative_path}") + + +def test_aggregator_batch_has_one_hook_and_one_batch_proposal(): + source = _source( + "backend/aggregator/agents/validator.py", + "ValidatorAgent", + "_assess_batch_quality", + ) + assert source.count("with_validator_hook(") == 1 + assert source.count("enqueue_optional_update(") == 1 + assert 'source_phase="batch_submission_validation"' in source + assert "batch=True" in source + assert source.index("submission_number") < source.index("enqueue_optional_update(") + + +def test_aggregator_cleanup_semantic_validators_are_integrated(): + cleanup = _source( + "backend/aggregator/agents/validator.py", + "ValidatorAgent", + "perform_cleanup_review", + ) + removal = _source( + "backend/aggregator/agents/validator.py", + "ValidatorAgent", + "validate_removal", + ) + + assert cleanup.count("with_validator_hook(") == 1 + assert cleanup.count("enqueue_optional_update(") == 1 + assert 'source_phase="cleanup_review_validation"' in cleanup + assert removal.count("with_validator_hook(") == 1 + assert removal.count("enqueue_optional_update(") == 1 + assert 'source_phase="cleanup_removal_validation"' in removal + + +def test_compiler_critique_semantic_validators_are_integrated(): + validation = _source( + "backend/compiler/core/compiler_coordinator.py", + "CompilerCoordinator", + "_validate_critique", + ) + cleanup = _source( + "backend/compiler/core/compiler_coordinator.py", + "CompilerCoordinator", + "_perform_critique_cleanup", + ) + + assert validation.count("with_validator_hook(") == 1 + assert validation.count("enqueue_optional_update(") == 1 + assert 'source_phase="critique_validation"' in validation + assert cleanup.count("with_validator_hook(") == 1 + assert cleanup.count("enqueue_optional_update(") == 1 + assert 'source_phase="critique_cleanup_validation"' in cleanup + assert validation.count("_retry") >= 1 + assert cleanup.count("_retry") >= 1 + + +def test_compiler_lean_placement_excludes_solution_path_proposals(): + validation = _source( + "backend/compiler/validation/compiler_validator.py", + "CompilerValidator", + "validate_submission", + ) + + assert "validation_mode = self._get_effective_validation_mode(submission)" in validation + assert validation.count('validation_mode != "rigor_lean_placement"') == 2 + assert validation.count("with_validator_hook(") == 1 + assert validation.count("enqueue_optional_update(") == 1 + + +def test_new_solver_surfaces_receive_only_budgeted_advisory_path(): + reference_expansion = _source( + "backend/autonomous/agents/reference_selector.py", + "ReferenceSelectorAgent", + "_request_expansion", + ) + reference_selection = _source( + "backend/autonomous/agents/reference_selector.py", + "ReferenceSelectorAgent", + "_make_final_selection", + ) + continuation = _source( + "backend/autonomous/core/autonomous_coordinator.py", + "AutonomousCoordinator", + "_brainstorm_continuation_decision", + ) + proof_identification = _source( + "backend/autonomous/agents/proof_identification_agent.py", + "ProofIdentificationAgent", + "identify_candidates", + ) + + for source in ( + reference_expansion, + reference_selection, + continuation, + proof_identification, + ): + assert "with_budgeted_solver_plan(" in source + assert "with_validator_hook(" not in source + assert "enqueue_optional_update(" not in source diff --git a/tests/regressions/test_workflow_solution_path.py b/tests/regressions/test_workflow_solution_path.py new file mode 100644 index 0000000..7a72922 --- /dev/null +++ b/tests/regressions/test_workflow_solution_path.py @@ -0,0 +1,187 @@ +from fastapi import FastAPI +from fastapi.testclient import TestClient + +from backend.api.routes.workflow import router +from backend.shared.solution_path import ( + ReviewDecision, + SolutionRoute, + SolutionPathManagerRegistry, +) + + +def test_solution_path_route_returns_typed_safe_fallback(): + app = FastAPI() + app.include_router(router) + + response = TestClient(app).get("/api/workflow/solution-path") + + assert response.status_code == 200 + payload = response.json() + assert payload["success"] is True + assert payload["enabled"] is False + assert payload["steps"] == [] + assert payload["ordering"] == "ordered" + assert payload["queued_proposals"] == 0 + assert payload["reviewing_proposals"] == 0 + assert payload["repair_required_proposals"] == 0 + assert isinstance(payload["message"], str) + + +def test_solution_path_schema_is_present_in_openapi(): + app = FastAPI() + app.include_router(router) + + operation = app.openapi()["paths"]["/api/workflow/solution-path"]["get"] + + assert operation["responses"]["200"]["content"]["application/json"]["schema"] + schema = app.openapi()["components"]["schemas"]["SolutionPathResponse"] + assert schema["properties"]["mode"]["enum"] == [ + "idle", + "aggregator", + "compiler", + "autonomous", + "leanoj", + ] + assert schema["properties"]["ordering"]["enum"] == ["ordered", "unordered"] + assert "queued_proposals" in schema["properties"] + assert "reviewing_proposals" in schema["properties"] + assert "lifecycle_generation" in schema["properties"] + assert "repair_required_proposals" in schema["properties"] + assert "repair_reason" in schema["properties"] + assert "/api/workflow/solution-path/resume" in app.openapi()["paths"] + + +def test_solution_path_route_exposes_loaded_manager_before_first_plan(monkeypatch, tmp_path): + import backend.shared.solution_path.registry as registry_module + import backend.shared.solution_path as solution_path_package + import backend.api.routes.workflow as workflow_module + + async def reviewer(proposal, plan): + return ReviewDecision(decision="approve") + + registry = SolutionPathManagerRegistry() + manager = __import__("asyncio").run( + registry.acquire( + tmp_path, + workflow_mode="manual", + user_prompt="Persisted prompt", + stable_run_id="manual", + reviewer=reviewer, + ) + ) + __import__("asyncio").run(manager.propose(SolutionRoute(steps=[]))) + __import__("asyncio").run(manager.stop()) + monkeypatch.setattr(registry_module, "solution_path_registry", registry) + monkeypatch.setattr(solution_path_package, "solution_path_registry", registry) + monkeypatch.setattr(workflow_module, "solution_path_registry", registry, raising=False) + + app = FastAPI() + app.include_router(router) + payload = TestClient(app).get("/api/workflow/solution-path").json() + + assert payload["run_id"] == "manual" + assert payload["queued_proposals"] == 1 + assert payload["pending_proposals"] == 1 + assert payload["steps"] == [] + + +def test_solution_path_route_selects_latest_of_multiple_resumable_runs( + monkeypatch, tmp_path +): + import asyncio + import backend.shared.solution_path.registry as registry_module + import backend.shared.solution_path as solution_path_package + + async def reviewer(proposal, plan): + return ReviewDecision(decision="approve") + + async def prepare(): + registry = SolutionPathManagerRegistry() + first = await registry.acquire( + tmp_path / "manual", + workflow_mode="manual", + user_prompt="First prompt", + stable_run_id="first-run", + reviewer=reviewer, + ) + await first.stop() + second = await registry.acquire( + tmp_path / "autonomous", + workflow_mode="autonomous", + user_prompt="Second prompt", + stable_run_id="second-run", + reviewer=reviewer, + ) + await second.stop() + return registry + + registry = asyncio.run(prepare()) + monkeypatch.setattr(registry_module, "solution_path_registry", registry) + monkeypatch.setattr(solution_path_package, "solution_path_registry", registry) + + app = FastAPI() + app.include_router(router) + payload = TestClient(app).get("/api/workflow/solution-path").json() + + assert payload["ownership"] == "resumable" + assert payload["mode"] == "autonomous" + assert payload["run_id"] == "second-run" + + +def test_resume_route_requeues_only_matching_generation(monkeypatch, tmp_path): + import asyncio + import backend.shared.solution_path.registry as registry_module + import backend.shared.solution_path as solution_path_package + + calls = 0 + + async def reviewer(proposal, plan): + nonlocal calls + calls += 1 + if calls == 1: + raise RuntimeError("No OpenRouter API key is available") + return ReviewDecision(decision="approve") + + async def prepare(): + registry = SolutionPathManagerRegistry() + manager = await registry.acquire( + tmp_path, + workflow_mode="manual", + user_prompt="Repair prompt", + stable_run_id="repair-run", + reviewer=reviewer, + ) + await manager.set_acceptance_count(5) + proposal = await manager.propose(SolutionRoute(steps=[])) + await manager.wait_idle() + await manager.stop() + return registry, manager, proposal + + registry, manager, proposal = asyncio.run(prepare()) + monkeypatch.setattr(registry_module, "solution_path_registry", registry) + monkeypatch.setattr(solution_path_package, "solution_path_registry", registry) + + app = FastAPI() + app.include_router(router) + client = TestClient(app) + generation = manager.state.lifecycle_generation + stale = client.post( + "/api/workflow/solution-path/resume", + json={ + "run_id": "repair-run", + "proposal_id": proposal.proposal_id, + "lifecycle_generation": generation - 1, + }, + ) + assert stale.status_code == 409 + + response = client.post( + "/api/workflow/solution-path/resume", + json={ + "run_id": "repair-run", + "proposal_id": proposal.proposal_id, + "lifecycle_generation": generation, + }, + ) + assert response.status_code == 200 + assert response.json()["proposal_status"] == "queued" diff --git a/tests/run_all_tests.py b/tests/run_all_tests.py index 66e8db2..51f5b18 100644 --- a/tests/run_all_tests.py +++ b/tests/run_all_tests.py @@ -2,13 +2,14 @@ Usage: python tests/run_all_tests.py - python tests/run_all_tests.py tests/test_build_info.py tests/test_proof_routes.py + python tests/run_all_tests.py tests/unit/test_build_info.py tests/workflow_scenarios/test_workflow_scenarios.py python tests/run_all_tests.py --stop-on-fail """ from __future__ import annotations import argparse +import shlex import subprocess import sys import time @@ -27,14 +28,27 @@ class TestFileResult: elapsed_seconds: float stdout: str stderr: str + allow_no_tests: bool = False @property def passed(self) -> bool: return self.returncode == 0 + @property + def skipped(self) -> bool: + return self.returncode == 5 and self.allow_no_tests + + @property + def successful(self) -> bool: + return self.passed or self.skipped + def discover_test_files() -> list[Path]: - return sorted(TESTS_DIR.glob("test_*.py")) + return sorted( + path + for path in TESTS_DIR.rglob("test_*.py") + if path.is_file() and "__pycache__" not in path.parts + ) def resolve_requested_files(requested_files: list[str]) -> list[Path]: @@ -55,7 +69,14 @@ def resolve_requested_files(requested_files: list[str]) -> list[Path]: if not path.exists(): raise SystemExit(f"Test file does not exist: {raw_path}") - if not path.is_file() or path.name == Path(__file__).name: + if ( + not path.is_file() + or path.parent == TESTS_DIR and path.name == Path(__file__).name + or TESTS_DIR not in path.parents + or not path.name.startswith("test_") + or path.suffix != ".py" + or "__pycache__" in path.parts + ): raise SystemExit(f"Not a runnable pytest file: {raw_path}") resolved.append(path) @@ -89,15 +110,20 @@ def run_test_file(test_file: Path, extra_pytest_args: list[str]) -> TestFileResu elapsed_seconds=elapsed_seconds, stdout=completed.stdout, stderr=completed.stderr, + allow_no_tests=any( + arg in {"-k", "-m", "--keyword", "--markers"} + or arg.startswith(("--keyword=", "--markers=")) + for arg in extra_pytest_args + ), ) def print_result(result: TestFileResult) -> None: - status = "PASS" if result.passed else "FAIL" + status = "PASS" if result.passed else "SKIP" if result.skipped else "FAIL" relative_path = result.path.relative_to(ROOT_DIR) print(f"[{status}] {relative_path} ({result.elapsed_seconds:.1f}s)") - if result.passed: + if result.successful: return output = "\n".join(part for part in (result.stdout, result.stderr) if part.strip()) @@ -111,12 +137,14 @@ def print_result(result: TestFileResult) -> None: def print_summary(results: list[TestFileResult]) -> None: passed = [result for result in results if result.passed] - failed = [result for result in results if not result.passed] + skipped = [result for result in results if result.skipped] + failed = [result for result in results if not result.successful] print("\n" + "=" * 80) print("TEST FILE SUMMARY") print("=" * 80) print(f"Passed: {len(passed)}") + print(f"Skipped: {len(skipped)}") print(f"Failed: {len(failed)}") print(f"Total: {len(results)}") @@ -141,7 +169,7 @@ def parse_args() -> argparse.Namespace: parser.add_argument( "test_files", nargs="*", - help="Optional test files to run. Defaults to every tests/test_*.py file.", + help="Optional test files to run. Defaults to every tests/**/test_*.py file.", ) parser.add_argument( "--stop-on-fail", @@ -156,11 +184,22 @@ def parse_args() -> argparse.Namespace: parser.add_argument( "--pytest-args", default="", - help="Additional arguments passed to pytest, split on spaces.", + help="Additional shell-style arguments passed to each pytest invocation.", ) return parser.parse_args() +def parse_pytest_args(raw_args: str) -> list[str]: + """Parse the quoted argument string supplied to ``--pytest-args``.""" + if not raw_args: + return [] + lexer = shlex.shlex(raw_args, posix=True) + lexer.whitespace_split = True + lexer.commenters = "" + lexer.escape = "" + return list(lexer) + + def main() -> int: args = parse_args() test_files = resolve_requested_files(args.test_files) @@ -171,10 +210,10 @@ def main() -> int: return 0 if not test_files: - print("No tests/test_*.py files found.") + print("No tests/**/test_*.py files found.") return 1 - extra_pytest_args = args.pytest_args.split() if args.pytest_args else [] + extra_pytest_args = parse_pytest_args(args.pytest_args) results: list[TestFileResult] = [] print(f"Running {len(test_files)} pytest file(s) separately...\n") @@ -183,11 +222,11 @@ def main() -> int: results.append(result) print_result(result) - if args.stop_on_fail and not result.passed: + if args.stop_on_fail and not result.successful: break print_summary(results) - return 0 if all(result.passed for result in results) else 1 + return 0 if all(result.successful for result in results) else 1 if __name__ == "__main__": diff --git a/tests/run_deep_workflows.py b/tests/run_deep_workflows.py new file mode 100644 index 0000000..cba8c03 --- /dev/null +++ b/tests/run_deep_workflows.py @@ -0,0 +1,46 @@ +"""Run the deterministic deep workflow suite on Windows and Unix-like systems.""" + +from __future__ import annotations + +import os +import subprocess +import sys +from pathlib import Path + + +ROOT_DIR = Path(__file__).resolve().parents[1] + + +def main() -> int: + env = os.environ.copy() + env["MOTO_WORKFLOW_DEEP_TESTS"] = "1" + model_result = subprocess.call( + [ + sys.executable, + "-m", + "pytest", + "tests/workflow_scenarios", + "tests/workflow_recombinatory", + "tests/workflow_cross_field", + ], + cwd=ROOT_DIR, + env=env, + ) + if model_result: + return model_result + + env["MOTO_REAL_ADAPTER_DEEP_TESTS"] = "1" + return subprocess.call( + [ + sys.executable, + "-m", + "pytest", + "tests/workflow_real_adapters/test_build_f_deep_matrix.py", + ], + cwd=ROOT_DIR, + env=env, + ) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_connectivity_status.py b/tests/test_connectivity_status.py deleted file mode 100644 index 7f35c61..0000000 --- a/tests/test_connectivity_status.py +++ /dev/null @@ -1,64 +0,0 @@ -from backend.api.routes.connectivity import _syntheticlib4_is_outdated - - -def test_syntheticlib4_outdated_when_cached_snapshot_has_expired_membership() -> None: - assert _syntheticlib4_is_outdated( - { - "credential_configured": True, - "auth_mode": "api_key", - "authenticated": True, - "membership_active": False, - }, - ready=True, - ) - - -def test_syntheticlib4_offline_ready_snapshot_is_not_outdated_without_live_access() -> None: - assert not _syntheticlib4_is_outdated( - { - "credential_configured": False, - "auth_mode": "built_in_offline_fixture", - "authenticated": True, - "membership_active": True, - }, - ready=True, - ) - - -def test_syntheticlib4_local_snapshot_with_explicit_membership_failure_is_outdated() -> None: - assert _syntheticlib4_is_outdated( - { - "credential_configured": False, - "auth_mode": "local_snapshot", - "authenticated": True, - "membership_active": False, - }, - ready=True, - ) - - -def test_syntheticlib4_expired_access_timestamp_is_outdated() -> None: - assert _syntheticlib4_is_outdated( - { - "credential_configured": True, - "auth_mode": "api_key", - "authenticated": True, - "membership_active": True, - "access_expires_at": "2000-01-01T00:00:00Z", - }, - ready=True, - ) - - -def test_syntheticlib4_future_access_timestamp_is_not_outdated() -> None: - assert not _syntheticlib4_is_outdated( - { - "credential_configured": True, - "auth_mode": "api_key", - "authenticated": True, - "membership_active": True, - "access_expires_at": "2999-01-01T00:00:00Z", - }, - ready=True, - ) - diff --git a/tests/test_proof_routes.py b/tests/test_proof_routes.py deleted file mode 100644 index 27ea2ec..0000000 --- a/tests/test_proof_routes.py +++ /dev/null @@ -1,112 +0,0 @@ -from datetime import datetime -from unittest import IsolatedAsyncioTestCase, TestCase, mock - -from fastapi import FastAPI -from fastapi.testclient import TestClient - -from backend.api.routes import proofs as proofs_route -from backend.shared.models import ProofRecord - - -class ManualProofScopeRouteTests(TestCase): - def setUp(self) -> None: - app = FastAPI() - app.include_router(proofs_route.router) - self.client = TestClient(app) - self._lean_enabled = proofs_route.system_config.lean4_enabled - proofs_route.system_config.lean4_enabled = False - - def tearDown(self) -> None: - proofs_route.system_config.lean4_enabled = self._lean_enabled - - def test_current_manual_proof_listing_uses_manual_database(self) -> None: - manual_db = mock.Mock() - manual_db.get_all_proofs = mock.AsyncMock(return_value=[]) - manual_db.count_proofs.return_value = {"total": 0, "novel": 0, "known": 0} - - with mock.patch.object(proofs_route, "manual_proof_database", manual_db): - response = self.client.get("/api/proofs?scope=manual") - - self.assertEqual(response.status_code, 200) - self.assertEqual(response.json()["scope"], "manual") - manual_db.get_all_proofs.assert_awaited_once_with() - - def test_manual_proof_library_uses_archived_history_only(self) -> None: - manual_db = mock.Mock() - manual_db.list_proof_library_from_history = mock.AsyncMock( - return_value=[ - { - "proof_id": "proof_history", - "session_id": "manual_proofs_2026-01-01_00-00-00", - "novel": True, - } - ] - ) - - with mock.patch.object(proofs_route, "manual_proof_database", manual_db): - response = self.client.get("/api/proofs/library?scope=manual") - - payload = response.json() - self.assertEqual(response.status_code, 200) - self.assertEqual(payload["scope"], "manual") - self.assertEqual(payload["counts"]["listed"], 1) - manual_db.list_proof_library_from_history.assert_awaited_once() - - def test_manual_certificate_routes_use_manual_database(self) -> None: - proof = ProofRecord( - proof_id="manual_proof_1", - theorem_statement="Manual theorem statement.", - theorem_name="manual_theorem", - source_type="brainstorm", - source_id="manual_aggregator", - source_title="Manual Aggregator", - lean_code="theorem manual_theorem : True := by trivial", - solver="Lean 4", - created_at=datetime(2026, 1, 1), - novel=True, - novelty_reasoning="Manual proof route test.", - attempt_count=1, - ) - manual_db = mock.Mock() - manual_db.get_proof = mock.AsyncMock(return_value=proof) - manual_db.get_lean_code = mock.AsyncMock(return_value=proof.lean_code) - - with mock.patch.object(proofs_route, "manual_proof_database", manual_db): - json_response = self.client.get("/api/proofs/manual_proof_1/certificate?scope=manual") - lean_response = self.client.get("/api/proofs/manual_proof_1/certificate.lean?scope=manual") - - self.assertEqual(json_response.status_code, 200) - self.assertEqual(json_response.json()["proof_id"], "manual_proof_1") - self.assertEqual(lean_response.status_code, 200) - self.assertIn("theorem manual_theorem", lean_response.text) - self.assertEqual(manual_db.get_proof.await_count, 2) - self.assertEqual(manual_db.get_lean_code.await_count, 2) - - -class ManualAggregatorProofEventLogTests(IsolatedAsyncioTestCase): - async def test_manual_aggregator_proof_event_is_broadcast_and_persisted_with_same_id(self) -> None: - with ( - mock.patch.object(proofs_route.websocket, "broadcast_event", new=mock.AsyncMock()) as broadcast, - mock.patch.object(proofs_route.event_log, "add_event", new=mock.AsyncMock()) as add_event, - ): - await proofs_route._broadcast_manual_aggregator_proof_event( - "proof_check_complete", - { - "source_type": "brainstorm", - "source_id": "manual_aggregator", - "verified_count": 1, - "novel_count": 0, - }, - ) - - broadcast.assert_awaited_once() - add_event.assert_awaited_once() - _, broadcast_payload = broadcast.await_args.args - _, message, persisted_payload = add_event.await_args.args - - self.assertEqual(message, "Proof check complete: 1 verified, 0 novel") - self.assertEqual( - broadcast_payload["manual_event_id"], - persisted_payload["manual_event_id"], - ) - self.assertEqual(persisted_payload["source_id"], "manual_aggregator") diff --git a/tests/test_runtime_root_isolation.py b/tests/test_runtime_root_isolation.py new file mode 100644 index 0000000..c078fa8 --- /dev/null +++ b/tests/test_runtime_root_isolation.py @@ -0,0 +1,146 @@ +import importlib +from pathlib import Path + +import pytest + +from backend.shared.config import bind_runtime_roots, system_config + + +@pytest.fixture +def restore_runtime_roots(): + original_data = system_config.data_dir + original_logs = system_config.logs_dir + yield + bind_runtime_roots(original_data, original_logs) + + +def test_bind_runtime_roots_rederives_complete_mutable_graph(tmp_path, restore_runtime_roots): + data_root = tmp_path / "instance-data" + logs_root = tmp_path / "instance-logs" + + before = system_config.runtime_root_generation + identity = bind_runtime_roots(data_root, logs_root) + + assert identity == ( + str(data_root.resolve()), + str(logs_root.resolve()), + before + 1, + ) + expected = { + system_config.user_uploads_dir: data_root / "user_uploads", + system_config.chroma_db_dir: data_root / "chroma_db", + system_config.shared_training_file: data_root / "rag_shared_training.txt", + system_config.compiler_outline_file: data_root / "compiler_outline.txt", + system_config.compiler_paper_file: data_root / "compiler_paper.txt", + system_config.auto_sessions_base_dir: data_root / "auto_sessions", + system_config.lean4_workspace_dir: data_root / "lean4_workspace", + } + for actual, wanted in expected.items(): + assert Path(actual) == wanted.resolve() + system_config.assert_path_in_data_root(actual) + + +def test_active_root_validator_rejects_stale_paths(tmp_path, restore_runtime_roots): + bind_runtime_roots(tmp_path / "active") + + with pytest.raises(RuntimeError, match="Stale mutable path"): + system_config.assert_path_in_data_root(tmp_path / "old" / "state.json") + + +def test_shared_training_rejects_stale_scoped_override(tmp_path, restore_runtime_roots): + from backend.aggregator.memory.shared_training import SharedTrainingMemory + + bind_runtime_roots(tmp_path / "old") + memory = SharedTrainingMemory() + memory.file_path = tmp_path / "old" / "brainstorms" / "topic.txt" + + bind_runtime_roots(tmp_path / "new") + with pytest.raises(RuntimeError, match="Stale shared-training override"): + _ = memory.file_path + + +@pytest.mark.asyncio +async def test_manual_and_compiler_stores_follow_rebind(tmp_path, restore_runtime_roots): + from backend.aggregator.core.coordinator import coordinator + from backend.aggregator.memory.event_log import event_log + from backend.aggregator.memory.shared_training import ( + load_manual_main_submitter_config, + save_manual_main_submitter_config, + shared_training_memory, + ) + from backend.compiler.memory.outline_memory import outline_memory + from backend.compiler.memory.paper_memory import paper_memory + + root_one = tmp_path / "one" + root_two = tmp_path / "two" + bind_runtime_roots(root_one) + await shared_training_memory.initialize() + await event_log.initialize() + await paper_memory.initialize() + await outline_memory.initialize() + await save_manual_main_submitter_config({"model_id": "first"}) + assert coordinator.stats_file_path == root_one.resolve() / "aggregator_stats.json" + + bind_runtime_roots(root_two) + await shared_training_memory.initialize() + await event_log.initialize() + await paper_memory.initialize() + await outline_memory.initialize() + await save_manual_main_submitter_config({"model_id": "second"}) + + assert shared_training_memory.file_path == root_two.resolve() / "rag_shared_training.txt" + assert event_log.file_path == root_two.resolve() / "aggregator_event_log.txt" + assert paper_memory.file_path == root_two.resolve() / "compiler_paper.txt" + assert outline_memory.file_path == root_two.resolve() / "compiler_outline.txt" + assert coordinator.stats_file_path == root_two.resolve() / "aggregator_stats.json" + assert await load_manual_main_submitter_config() == {"model_id": "second"} + assert (root_one / "manual_main_submitter_config.json").exists() + + +def test_rag_manager_import_and_construction_are_side_effect_free( + tmp_path, restore_runtime_roots +): + bind_runtime_roots(tmp_path / "rag-data") + module = importlib.import_module("backend.aggregator.core.rag_manager") + manager = module.RAGManager() + + assert manager.is_initialized is False + assert not Path(system_config.chroma_db_dir).exists() + + +@pytest.mark.asyncio +async def test_rag_manager_reopens_under_active_root(tmp_path, restore_runtime_roots): + from backend.aggregator.core.rag_manager import RAGManager + + manager = RAGManager() + first_root = tmp_path / "rag-one" + second_root = tmp_path / "rag-two" + try: + bind_runtime_roots(first_root) + await manager.ensure_initialized() + first_identity = manager._root_identity + assert (first_root / "chroma_db").exists() + + bind_runtime_roots(second_root) + await manager.ensure_initialized() + assert manager._root_identity != first_identity + assert manager._root_identity == system_config.runtime_root_identity() + assert (second_root / "chroma_db").exists() + finally: + await manager.reset() + + +def test_default_proof_stores_follow_rebind(tmp_path, restore_runtime_roots): + from backend.autonomous.memory.proof_database import ( + manual_proof_database, + proof_database, + ) + + bind_runtime_roots(tmp_path / "proof-data") + + assert proof_database._get_index_path().parent == ( + tmp_path / "proof-data" / "proofs" + ).resolve() + assert manual_proof_database._get_index_path().parent == ( + tmp_path / "proof-data" / "manual_proofs" + ).resolve() diff --git a/tests/test_runtime_root_lock.py b/tests/test_runtime_root_lock.py new file mode 100644 index 0000000..fcda609 --- /dev/null +++ b/tests/test_runtime_root_lock.py @@ -0,0 +1,63 @@ +from __future__ import annotations + +import os +from pathlib import Path +import subprocess +import sys + +import pytest + +from backend.shared.runtime_root_lock import RuntimeRootInUseError, RuntimeRootLease + + +def test_runtime_root_lease_rejects_same_process_owner(tmp_path: Path) -> None: + first = RuntimeRootLease(tmp_path).acquire() + try: + with pytest.raises(RuntimeRootInUseError): + RuntimeRootLease(tmp_path).acquire() + finally: + first.release() + first.release() + + replacement = RuntimeRootLease(tmp_path).acquire() + replacement.release() + + +def test_runtime_root_lease_allows_distinct_roots(tmp_path: Path) -> None: + first = RuntimeRootLease(tmp_path / "one").acquire() + second = RuntimeRootLease(tmp_path / "two").acquire() + second.release() + first.release() + + +@pytest.mark.skipif(os.name != "nt", reason="Windows lock regression") +def test_runtime_root_lease_rejects_competing_process_then_recovers(tmp_path: Path) -> None: + script = """ +import sys +from backend.shared.runtime_root_lock import RuntimeRootLease +lease = RuntimeRootLease(sys.argv[1]).acquire() +print("LOCKED", flush=True) +sys.stdin.readline() +lease.release() +""" + child = subprocess.Popen( + [sys.executable, "-c", script, str(tmp_path)], + cwd=Path(__file__).resolve().parents[1], + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + ) + try: + assert child.stdout is not None + assert child.stdout.readline().strip() == "LOCKED" + with pytest.raises(RuntimeRootInUseError): + RuntimeRootLease(tmp_path).acquire() + finally: + if child.stdin is not None: + child.stdin.write("\n") + child.stdin.flush() + child.wait(timeout=10) + + recovered = RuntimeRootLease(tmp_path).acquire() + recovered.release() diff --git a/tests/test_sleep_inhibitor.py b/tests/test_sleep_inhibitor.py new file mode 100644 index 0000000..4e6ef4e --- /dev/null +++ b/tests/test_sleep_inhibitor.py @@ -0,0 +1,148 @@ +from __future__ import annotations + +import threading +import time + +from backend.shared.sleep_inhibitor import ( + ES_CONTINUOUS, + ES_SYSTEM_REQUIRED, + SleepInhibitor, +) + + +def test_windows_inhibitor_is_owner_idempotent_and_uses_one_worker_thread(monkeypatch): + monkeypatch.setattr( + "backend.shared.sleep_inhibitor.system_config.generic_mode", + False, + ) + calls: list[tuple[int, int]] = [] + + def setter(flags: int) -> int: + calls.append((flags, threading.get_ident())) + return 1 + + inhibitor = SleepInhibitor( + platform="win32", + execution_state_setter=setter, + ) + + inhibitor.acquire("aggregator") + inhibitor.acquire("aggregator") + inhibitor.acquire("compiler") + inhibitor.release("aggregator") + + deadline = time.monotonic() + 2 + while len(calls) < 1 and time.monotonic() < deadline: + time.sleep(0.01) + assert [flags for flags, _ in calls] == [ES_CONTINUOUS | ES_SYSTEM_REQUIRED] + assert inhibitor.owners == frozenset({"compiler"}) + + inhibitor.release("compiler") + + deadline = time.monotonic() + 2 + while len(calls) < 2 and time.monotonic() < deadline: + time.sleep(0.01) + assert [flags for flags, _ in calls] == [ + ES_CONTINUOUS | ES_SYSTEM_REQUIRED, + ES_CONTINUOUS, + ] + assert len({thread_id for _, thread_id in calls}) == 1 + assert inhibitor.owners == frozenset() + + +def test_non_windows_and_generic_mode_are_noops(monkeypatch): + calls: list[int] = [] + setter = lambda flags: calls.append(flags) or 1 + + monkeypatch.setattr( + "backend.shared.sleep_inhibitor.system_config.generic_mode", + False, + ) + non_windows = SleepInhibitor(platform="linux", execution_state_setter=setter) + non_windows.acquire("autonomous") + non_windows.release_all() + + monkeypatch.setattr( + "backend.shared.sleep_inhibitor.system_config.generic_mode", + True, + ) + generic = SleepInhibitor(platform="win32", execution_state_setter=setter) + generic.acquire("leanoj") + generic.release_all() + + assert calls == [] + assert non_windows.owners == frozenset() + assert generic.owners == frozenset() + + +def test_native_failure_does_not_lose_logical_owner(monkeypatch): + monkeypatch.setattr( + "backend.shared.sleep_inhibitor.system_config.generic_mode", + False, + ) + + def failing_setter(_flags: int) -> int: + raise OSError("simulated power API failure") + + inhibitor = SleepInhibitor( + platform="win32", + execution_state_setter=failing_setter, + ) + + inhibitor.acquire("autonomous") + assert inhibitor.owners == frozenset({"autonomous"}) + + inhibitor.release_all() + assert inhibitor.owners == frozenset() + + +def test_public_updates_do_not_wait_for_native_setter(monkeypatch): + monkeypatch.setattr( + "backend.shared.sleep_inhibitor.system_config.generic_mode", + False, + ) + entered = threading.Event() + release = threading.Event() + + def blocked_setter(_flags: int) -> int: + entered.set() + release.wait(timeout=2) + return 1 + + inhibitor = SleepInhibitor( + platform="win32", + execution_state_setter=blocked_setter, + ) + started = time.monotonic() + inhibitor.acquire("aggregator") + assert time.monotonic() - started < 0.2 + assert entered.wait(timeout=1) + + started = time.monotonic() + inhibitor.release("aggregator") + assert time.monotonic() - started < 0.2 + release.set() + + +def test_return_zero_retries_until_native_acquire_succeeds(monkeypatch): + monkeypatch.setattr( + "backend.shared.sleep_inhibitor.system_config.generic_mode", + False, + ) + calls = 0 + + def setter(_flags: int) -> int: + nonlocal calls + calls += 1 + return 0 if calls == 1 else 1 + + monkeypatch.setattr("backend.shared.sleep_inhibitor.time.sleep", lambda _delay: None) + inhibitor = SleepInhibitor(platform="win32", execution_state_setter=setter) + inhibitor.acquire("autonomous") + + deadline = time.monotonic() + 2 + while not inhibitor.native_active and time.monotonic() < deadline: + time.sleep(0.01) + assert calls >= 2 + assert inhibitor.native_active is True + inhibitor.release_all() diff --git a/tests/test_workflow_start_guard_ownership.py b/tests/test_workflow_start_guard_ownership.py new file mode 100644 index 0000000..1097821 --- /dev/null +++ b/tests/test_workflow_start_guard_ownership.py @@ -0,0 +1,321 @@ +from __future__ import annotations + +import asyncio +import threading +from types import SimpleNamespace + +import pytest + +from backend.aggregator.core.rag_manager import RAGManager +from backend.shared.rag_lock import RAGOperationLock +from backend.shared.workflow_start_guard import WorkflowStartGuard + + +def test_workflow_guard_commits_and_releases_one_owner(monkeypatch): + calls: list[tuple[str, str | None]] = [] + monkeypatch.setattr( + "backend.shared.workflow_start_guard.sleep_inhibitor.acquire", + lambda owner: calls.append(("acquire", owner)), + ) + monkeypatch.setattr( + "backend.shared.workflow_start_guard.sleep_inhibitor.release", + lambda owner: calls.append(("release", owner)), + ) + + guard = WorkflowStartGuard() + first_lease = guard.commit("manual_aggregator") + assert guard.commit("manual_aggregator") == first_lease + assert guard.release("manual_aggregator") is False + guard.release(first_lease) + + assert guard.active_owner is None + assert calls == [ + ("acquire", first_lease), + ("release", first_lease), + ] + + +def test_workflow_guard_rejects_different_committed_owner(monkeypatch): + monkeypatch.setattr( + "backend.shared.workflow_start_guard.sleep_inhibitor.acquire", + lambda _owner: None, + ) + guard = WorkflowStartGuard() + guard.commit("manual_compiler_proof_only") + + try: + guard.commit("autonomous") + except RuntimeError as exc: + assert "manual_compiler_proof_only" in str(exc) + else: + raise AssertionError("Expected a conflicting owner to be rejected") + + +def test_stale_lease_cannot_release_new_generation(monkeypatch): + monkeypatch.setattr( + "backend.shared.workflow_start_guard.sleep_inhibitor.acquire", + lambda _owner: None, + ) + monkeypatch.setattr( + "backend.shared.workflow_start_guard.sleep_inhibitor.release", + lambda _owner: None, + ) + guard = WorkflowStartGuard() + stale = guard.commit("autonomous") + assert guard.release(stale) is True + current = guard.commit("autonomous") + + assert current.generation > stale.generation + assert guard.release(stale) is False + assert guard.active_lease == current + + +def test_inhibitor_setup_failure_does_not_fail_logical_commit(monkeypatch): + def fail(_owner): + raise RuntimeError("simulated worker setup failure") + + monkeypatch.setattr( + "backend.shared.workflow_start_guard.sleep_inhibitor.acquire", + fail, + ) + guard = WorkflowStartGuard() + + lease = guard.commit("leanoj") + + assert guard.active_lease == lease + assert guard.active_owner == "leanoj" + + +def test_inhibitor_release_failure_does_not_fail_logical_release( + monkeypatch, caplog +): + def fail(_lease): + raise RuntimeError("simulated worker release failure") + + monkeypatch.setattr( + "backend.shared.workflow_start_guard.sleep_inhibitor.acquire", + lambda _lease: None, + ) + monkeypatch.setattr( + "backend.shared.workflow_start_guard.sleep_inhibitor.release", + fail, + ) + guard = WorkflowStartGuard() + lease = guard.commit("manual_aggregator") + + assert guard.release(lease) is True + assert guard.active_owner is None + assert "Unable to release desktop sleep inhibition for manual_aggregator" in caplog.text + + +@pytest.mark.asyncio +async def test_workflow_guard_serializes_start_and_stop_lifecycle_work(): + guard = WorkflowStartGuard() + start_entered = asyncio.Event() + allow_start_to_finish = asyncio.Event() + stop_entered = asyncio.Event() + + async def run_start() -> None: + async with guard.reserve(): + start_entered.set() + await allow_start_to_finish.wait() + + async def run_stop() -> None: + async with guard.reserve(): + stop_entered.set() + + start_task = asyncio.create_task(run_start()) + await start_entered.wait() + stop_task = asyncio.create_task(run_stop()) + await asyncio.sleep(0) + + assert not stop_entered.is_set() + + allow_start_to_finish.set() + await asyncio.gather(start_task, stop_task) + + assert stop_entered.is_set() + + +def test_release_all_invalidates_same_owner_generation(monkeypatch): + monkeypatch.setattr( + "backend.shared.workflow_start_guard.sleep_inhibitor.acquire", + lambda _lease: None, + ) + monkeypatch.setattr( + "backend.shared.workflow_start_guard.sleep_inhibitor.release_all", + lambda: None, + ) + guard = WorkflowStartGuard() + stale = guard.commit("leanoj") + + guard.release_all() + current = guard.commit("leanoj") + + assert current.generation > stale.generation + assert guard.release(stale) is False + assert guard.active_lease == current + + +@pytest.mark.asyncio +async def test_cancelled_chroma_call_finishes_worker_before_releasing_lock(monkeypatch): + manager = RAGManager() + rag_lock = RAGOperationLock() + worker_entered = threading.Event() + allow_worker_to_finish = threading.Event() + second_operation_entered = asyncio.Event() + + monkeypatch.setattr( + "backend.aggregator.core.rag_manager.rag_operation_lock", + rag_lock, + ) + monkeypatch.setattr(manager, "_ensure_initialized_locked", lambda: None) + + def blocking_native_call() -> str: + worker_entered.set() + allow_worker_to_finish.wait() + return "done" + + manager.collections[256] = SimpleNamespace(run=blocking_native_call) + first = asyncio.create_task( + manager._run_chroma_call("first", 256, "run") + ) + await asyncio.to_thread(worker_entered.wait) + first.cancel() + + async def run_second() -> None: + await rag_lock.acquire("second") + try: + second_operation_entered.set() + finally: + rag_lock.release() + + second = asyncio.create_task(run_second()) + await asyncio.sleep(0) + assert not second_operation_entered.is_set() + + allow_worker_to_finish.set() + with pytest.raises(asyncio.CancelledError): + await first + await second + assert second_operation_entered.is_set() + assert rag_lock._acquisition_count == 0 + + +@pytest.mark.asyncio +async def test_cancelled_waiter_cannot_release_current_rag_owner(): + rag_lock = RAGOperationLock() + owner_release = asyncio.Event() + waiter_started = asyncio.Event() + + async def owner() -> None: + async with rag_lock.operation("owner"): + waiter_started.set() + await owner_release.wait() + + async def waiter() -> None: + async with rag_lock.operation("waiter"): + raise AssertionError("cancelled waiter entered the critical section") + + owner_task = asyncio.create_task(owner()) + await waiter_started.wait() + waiter_task = asyncio.create_task(waiter()) + await asyncio.sleep(0) + waiter_task.cancel() + with pytest.raises(asyncio.CancelledError): + await waiter_task + + assert rag_lock._lock.locked() + owner_release.set() + await owner_task + assert not rag_lock._lock.locked() + + +@pytest.mark.asyncio +async def test_chroma_method_is_resolved_after_lifecycle_initialization(monkeypatch): + manager = RAGManager() + stale = SimpleNamespace(query=lambda **_kwargs: "stale") + current = SimpleNamespace(query=lambda **_kwargs: "current") + manager.collections[256] = stale + + def initialize() -> None: + manager.collections[256] = current + + async def direct_native(_name, func, *args, **kwargs): + return func(*args, **kwargs) + + monkeypatch.setattr(manager, "_ensure_initialized_locked", initialize) + monkeypatch.setattr(manager, "_await_native_worker", direct_native) + + result = await manager._run_chroma_call("query", 256, "query") + + assert result == "current" + + +@pytest.mark.asyncio +async def test_failed_chroma_upsert_does_not_commit_memory(monkeypatch): + manager = RAGManager() + existing = SimpleNamespace(chunk_id="existing") + incoming = SimpleNamespace( + chunk_id="incoming", + text="new text", + metadata={"source_file": "source.txt"}, + ) + manager.chunks_by_size[256] = [existing] + manager.collections[256] = SimpleNamespace( + upsert=lambda **_kwargs: (_ for _ in ()).throw(RuntimeError("write failed")) + ) + monkeypatch.setattr(manager, "_ensure_initialized_locked", lambda: None) + + async def direct_native(_name, func, *args, **kwargs): + return func(*args, **kwargs) + + monkeypatch.setattr(manager, "_await_native_worker", direct_native) + monkeypatch.setattr( + "backend.aggregator.core.rag_manager.api_client_manager.get_embeddings", + lambda _texts: asyncio.sleep(0, result=[[0.0, 1.0]]), + ) + + with pytest.raises(RuntimeError, match="write failed"): + await manager._add_chunks([incoming], 256) + + assert manager.chunks_by_size[256] == [existing] + + +@pytest.mark.asyncio +async def test_failed_chunk_cap_delete_preserves_chunks_and_embeddings(monkeypatch): + manager = RAGManager() + first = SimpleNamespace( + chunk_id="first", + is_permanent=False, + embedding=[1.0, 0.0], + ) + second = SimpleNamespace( + chunk_id="second", + is_permanent=False, + embedding=[0.0, 1.0], + ) + manager.chunks_by_size[256] = [first, second] + sentinel = object() + manager.bm25_index[256] = sentinel + monkeypatch.setattr( + "backend.aggregator.core.rag_manager.rag_config.submitter_chunk_intervals", + [256], + ) + monkeypatch.setattr( + "backend.aggregator.core.rag_manager.rag_config.max_chunks_per_size", + 1, + ) + + async def fail_delete(*_args, **_kwargs): + raise RuntimeError("delete failed") + + monkeypatch.setattr(manager, "_run_chroma_call", fail_delete) + + with pytest.raises(RuntimeError, match="delete failed"): + await manager._enforce_chunk_cap() + + assert manager.chunks_by_size[256] == [first, second] + assert first.embedding == [1.0, 0.0] + assert second.embedding == [0.0, 1.0] + assert manager.bm25_index[256] is sentinel diff --git a/tests/unit/__init__.py b/tests/unit/__init__.py new file mode 100644 index 0000000..43f2143 --- /dev/null +++ b/tests/unit/__init__.py @@ -0,0 +1,2 @@ +"""Focused unit tests may be moved here in small batches.""" + diff --git a/tests/unit/test_all_purpose_aggregator_prompts.py b/tests/unit/test_all_purpose_aggregator_prompts.py new file mode 100644 index 0000000..d9af450 --- /dev/null +++ b/tests/unit/test_all_purpose_aggregator_prompts.py @@ -0,0 +1,377 @@ +import pytest +from unittest.mock import AsyncMock + +from backend.aggregator.agents import submitter as submitter_module +from backend.aggregator.agents.submitter import SubmitterAgent, _requires_lean_retry_schema +from backend.shared.brainstorm_proof_gate import BrainstormProofGateResult +from backend.aggregator.prompts.submitter_prompts import ( + CREATIVITY_EMPHASIS_BOOST_PROMPT, + build_submitter_prompt, + get_submitter_json_schema, + get_submitter_system_prompt, +) +from backend.aggregator.prompts.validator_prompts import ( + get_cleanup_review_system_prompt, + get_removal_validation_system_prompt, + get_validator_dual_json_schema, + get_validator_dual_system_prompt, + get_validator_system_prompt, + get_validator_triple_json_schema, + get_validator_triple_system_prompt, +) + + +def test_ordinary_submitter_seeks_direct_all_purpose_solutions() -> None: + prompt = get_submitter_system_prompt(lean4_enabled=False) + + assert "strongest credible and genuinely novel solution" in prompt + assert "user's exact objective" in prompt + assert "WHOLE question" in prompt + assert "next best necessary piece" in prompt + for contribution in ( + "invention", + "mechanism", + "design", + "algorithm", + "experimental proposal", + "falsifiable hypothesis", + "counterexample", + "impossibility argument", + "implementation strategy", + "risk analysis", + ): + assert contribution in prompt + assert "You are a mathematical submitter" not in prompt + assert "ALL submissions must be rooted in sound mathematical reasoning" not in prompt + + +def test_mathematics_and_lean_remain_affirmatively_available() -> None: + prompt = get_submitter_system_prompt(lean4_enabled=True) + schema = get_submitter_json_schema(lean4_enabled=True) + + assert "Mathematical reasoning and formal proof remain first-class" in prompt + assert "Mathematics, theorem discovery, proof, and formalization are explicitly welcome" in prompt + assert "mathematical claims require sound derivation, proof, or explicit assumptions" in prompt + for field in ( + "theorem_statement", + "formal_sketch", + "expected_novelty_tier", + "prompt_relevance_rationale", + "novelty_rationale", + "why_not_standard_known_result", + "theorem_name", + "lean_code", + "reasoning", + ): + assert field in schema + assert "routine helpers" in schema + assert "trivial/easy proofs" in schema + + +def test_lean_schema_is_absent_when_disabled() -> None: + prompt = build_submitter_prompt("objective", "context", lean4_enabled=False) + + for lean_only_text in ( + "lean_proof", + "Lean proof candidate", + "theorem_statement", + "formal_sketch", + "expected_novelty_tier", + "prompt_relevance_rationale", + "novelty_rationale", + "why_not_standard_known_result", + "theorem_name", + "lean_code", + ): + assert lean_only_text not in prompt + + +def test_claim_type_rigor_covers_engineering_and_provenance() -> None: + prompt = get_submitter_system_prompt() + + for requirement in ( + "concrete mechanism", + "constraints", + "feasibility reasoning", + "failure modes", + "verification plan", + "Literature claims", + "Empirical claims", + "Artifact claims", + ): + assert requirement in prompt + assert "NEVER invent experiments" in prompt + assert "proposed experiment" in prompt + assert "without claiming that a prototype or test result already exists" in get_submitter_json_schema() + + +@pytest.mark.parametrize( + "prompt_builder", + [ + get_validator_system_prompt, + get_validator_dual_system_prompt, + get_validator_triple_system_prompt, + ], +) +def test_all_validator_batch_sizes_share_the_same_policy(prompt_builder) -> None: + prompt = prompt_builder() + + for criterion in ( + "SHARED ALL-PURPOSE EVALUATION CRITERIA", + "Direct impact", + "Genuine novelty", + "Correctness", + "Provenance and uncertainty", + "Specificity and verification", + "Feasibility", + "Non-redundancy", + "Mathematics remains first-class", + "non-mathematical work must not be rejected merely for lacking theorem or proof form", + ): + assert criterion in prompt + assert "user's exact objective" in prompt + assert "[LEAN 4 VERIFIED BRAINSTORM PROOF]" in prompt + assert "Lean verification only establishes formal validity" in prompt + for forbidden_universal in ( + "evaluate mathematical submissions", + "user's mathematical prompt", + "rather than mathematical solutions", + "lacks mathematical rigor", + "invalid proof structure", + ): + assert forbidden_universal not in prompt + + +def test_batch_validators_preserve_independent_redundancy_resolution() -> None: + dual = get_validator_dual_system_prompt() + triple = get_validator_triple_system_prompt() + + assert "TWO SEPARATE, INDEPENDENT decisions first" in dual + assert "Keep ONLY the stronger/more complete one" in dual + assert "THREE SEPARATE, INDEPENDENT decisions first" in triple + assert "keep ONLY the strongest" in triple + + +def test_cleanup_and_removal_preserve_unique_non_mathematical_value() -> None: + cleanup = get_cleanup_review_system_prompt() + removal = get_removal_validation_system_prompt() + + assert "AT MOST ONE" in cleanup + assert "unique mechanism, design, algorithm, evidence plan, risk analysis" in cleanup + assert "select the WEAKEST one for removal" in cleanup + assert "When in doubt, DO NOT recommend removal" in cleanup + assert "ANY unique value" in removal + assert "If uncertain, REJECT the removal" in removal + assert "unique value not covered elsewhere" in removal + for prompt in (cleanup, removal): + assert "[LEAN 4 VERIFIED BRAINSTORM PROOF]" in prompt + assert "Lean verification only establishes formal validity" in prompt + assert "Do NOT re-litigate Lean syntax" in prompt + + +def test_validator_examples_use_claim_appropriate_mixed_domain_standards() -> None: + dual = get_validator_dual_json_schema() + triple = get_validator_triple_json_schema() + + assert "segmented battery-isolation mechanism" in dual + assert "complete mathematical argument" in dual + assert "software isolation design" in triple + assert "rigorous counterexample" in triple + assert "provide a concrete mechanism and verification plan" in triple + assert "provide a test that could falsify the claim" in triple + + +def test_retry_variant_detection_preserves_recognizable_lean_candidates() -> None: + malformed_lean_outputs = ( + '{"submission_type": "lean_proof", "lean_code": "theorem x', + '{"theorem_statement": "High-impact result", "formal_sketch": ', + '{"lean_code": "import Mathlib\\ntheorem target : True := by trivial"', + ) + + for output in malformed_lean_outputs: + assert _requires_lean_retry_schema(output, lean4_enabled=True) + assert not _requires_lean_retry_schema(output, lean4_enabled=False) + assert not _requires_lean_retry_schema( + '{"submission_type": "idea", "submission": "engineering mechanism"', + lean4_enabled=True, + ) + + +def test_retry_source_requires_complete_schema_and_forbids_lean_downgrade() -> None: + import inspect + + from backend.aggregator.agents import submitter as submitter_module + + source = inspect.getsource(submitter_module.SubmitterAgent._generate_submission) + assert "get_submitter_json_schema" in source + assert "do not convert it into an ordinary idea" in source + assert '"your submission (LaTeX allowed' not in source + + +def _completion(content: str) -> dict: + return {"choices": [{"message": {"content": content}}]} + + +def _patch_submitter_dependencies(monkeypatch, agent: SubmitterAgent, responses: list[dict]) -> None: + monkeypatch.setattr( + submitter_module.shared_training_memory, + "get_all_content", + AsyncMock(return_value=""), + ) + monkeypatch.setattr(agent.local_memory, "get_all_content", AsyncMock(return_value="")) + monkeypatch.setattr(agent.local_memory, "add_rejection", AsyncMock()) + monkeypatch.setattr( + submitter_module.context_allocator, + "allocate_submitter_context", + AsyncMock(return_value={"direct": "", "rag_context": None}), + ) + monkeypatch.setattr( + submitter_module.api_client_manager, + "prewarm_assistant_memory_context", + AsyncMock(), + ) + monkeypatch.setattr( + submitter_module.api_client_manager, + "generate_completion", + AsyncMock(side_effect=responses), + ) + monkeypatch.setattr( + submitter_module.api_client_manager, + "extract_call_metadata", + lambda response: {}, + ) + monkeypatch.setattr( + submitter_module.lm_studio_client, + "cache_model_load_config", + AsyncMock(), + ) + + +@pytest.mark.asyncio +async def test_ordinary_engineering_idea_parses_without_proof_gate(monkeypatch) -> None: + agent = SubmitterAgent( + submitter_id=1, + model_name="model-a", + user_prompt="Design a safer battery pack", + user_files_content={}, + context_window=32_000, + max_output_tokens=2_000, + ) + _patch_submitter_dependencies( + monkeypatch, + agent, + [ + _completion( + '{"submission_type":"idea","submission":"Use segmented isolation with ' + 'redundant contactors, explicit sensor-failure handling, and staged ' + 'fault-injection tests.","reasoning":"This is a concrete mechanism ' + 'with constraints and verification."}' + ) + ], + ) + proof_gate = AsyncMock() + monkeypatch.setattr(submitter_module, "verify_brainstorm_proof_candidate", proof_gate) + monkeypatch.setattr(submitter_module.system_config, "lean4_enabled", True) + + submission = await agent._generate_submission() + + assert submission is not None + assert "segmented isolation" in submission.content + proof_gate.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_malformed_lean_retry_still_passes_proof_gate(monkeypatch) -> None: + agent = SubmitterAgent( + submitter_id=1, + model_name="model-a", + user_prompt="Prove the target theorem", + user_files_content={}, + context_window=32_000, + max_output_tokens=2_000, + ) + repaired = ( + '{"submission_type":"lean_proof","theorem_statement":"True",' + '"formal_sketch":"Use True.intro.","expected_novelty_tier":"novel_formulation",' + '"prompt_relevance_rationale":"Directly proves the requested target.",' + '"novelty_rationale":"A prompt-specific formulation.",' + '"why_not_standard_known_result":"The exact target is prompt-specific.",' + '"theorem_name":"target","lean_code":"theorem target : True := by trivial",' + '"reasoning":"Complete Lean proof candidate."}' + ) + _patch_submitter_dependencies( + monkeypatch, + agent, + [ + _completion('{"submission_type":"lean_proof","lean_code":"theorem target'), + _completion(repaired), + ], + ) + gate_result = BrainstormProofGateResult( + accepted=True, + submission_content="[LEAN 4 VERIFIED BRAINSTORM PROOF]\nTrue", + theorem_statement="True", + theorem_name="target", + formal_sketch="Use True.intro.", + expected_novelty_tier="novel_formulation", + prompt_relevance_rationale="Directly proves the target.", + novelty_rationale="Prompt-specific formulation.", + why_not_standard_known_result="Exact prompt target.", + lean_code="theorem target : True := by trivial", + reasoning="Lean verified.", + ) + proof_gate = AsyncMock(return_value=gate_result) + monkeypatch.setattr(submitter_module, "verify_brainstorm_proof_candidate", proof_gate) + monkeypatch.setattr(submitter_module.system_config, "lean4_enabled", True) + + submission = await agent._generate_submission() + + assert submission is not None + proof_gate.assert_awaited_once() + assert submission.content.startswith("[LEAN 4 VERIFIED BRAINSTORM PROOF]") + assert submission.metadata["brainstorm_lean_proof"]["lean_code"].startswith("theorem target") + + +@pytest.mark.asyncio +async def test_rejected_proof_gate_cannot_emit_ordinary_submission(monkeypatch) -> None: + agent = SubmitterAgent( + submitter_id=1, + model_name="model-a", + user_prompt="Prove the target theorem", + user_files_content={}, + context_window=32_000, + max_output_tokens=2_000, + ) + candidate = ( + '{"submission_type":"lean_proof","theorem_statement":"True",' + '"formal_sketch":"Use True.intro.","expected_novelty_tier":"novel_formulation",' + '"prompt_relevance_rationale":"Direct target.","novelty_rationale":"Prompt-specific.",' + '"why_not_standard_known_result":"Exact prompt target.","theorem_name":"target",' + '"lean_code":"theorem target : True := by trivial","reasoning":"Candidate."}' + ) + _patch_submitter_dependencies(monkeypatch, agent, [_completion(candidate)]) + proof_gate = AsyncMock( + return_value=BrainstormProofGateResult( + accepted=False, + failure_feedback="Rejected by the protected proof gate.", + ) + ) + monkeypatch.setattr(submitter_module, "verify_brainstorm_proof_candidate", proof_gate) + monkeypatch.setattr(submitter_module.system_config, "lean4_enabled", True) + + submission = await agent._generate_submission() + + assert submission is None + proof_gate.assert_awaited_once() + agent.local_memory.add_rejection.assert_awaited_once() + + +def test_creativity_emphasis_is_optional_rigorous_and_non_fabricating() -> None: + normal = build_submitter_prompt("objective", "context", creativity_emphasized=False) + creative = build_submitter_prompt("objective", "context", creativity_emphasized=True) + + assert "CREATIVITY EMPHASIS BOOST" not in normal + assert CREATIVITY_EMPHASIS_BOOST_PROMPT in creative + assert "near-solution or adjacent solution" in creative + assert "not permission to fabricate" in creative + assert "same JSON schema and rigor requirements" in creative diff --git a/tests/unit/test_all_purpose_autonomous_prompts.py b/tests/unit/test_all_purpose_autonomous_prompts.py new file mode 100644 index 0000000..e573ef6 --- /dev/null +++ b/tests/unit/test_all_purpose_autonomous_prompts.py @@ -0,0 +1,587 @@ +import importlib +import re +from types import SimpleNamespace +from unittest.mock import AsyncMock + +import pytest + +from backend.autonomous.core.autonomous_coordinator import AutonomousCoordinator +from backend.autonomous.prompts.completion_prompts import ( + build_completion_review_prompt, + build_completion_self_validation_prompt, + get_completion_review_json_schema, + get_completion_review_system_prompt, + get_completion_self_validation_json_schema, + get_completion_self_validation_system_prompt, +) +from backend.autonomous.prompts.paper_continuation_prompts import ( + build_continuation_decision_prompt, + build_continuation_validation_prompt, + get_continuation_decision_json_schema, + get_continuation_decision_system_prompt, + get_continuation_validator_json_schema, + get_continuation_validator_system_prompt, +) +from backend.autonomous.prompts.paper_reference_prompts import ( + build_additional_reference_expansion_prompt, + build_pre_brainstorm_expansion_prompt, + build_reference_expansion_prompt, + build_reference_selection_prompt, + get_additional_reference_expansion_system_prompt, + get_pre_brainstorm_expansion_system_prompt, + get_reference_expansion_json_schema, + get_reference_expansion_system_prompt, + get_reference_selection_json_schema, + get_reference_selection_system_prompt, +) +from backend.autonomous.prompts.paper_title_exploration_prompts import ( + build_title_exploration_user_prompt, +) +from backend.autonomous.prompts.paper_title_prompts import ( + build_paper_title_prompt, + build_paper_title_validation_prompt, + get_paper_title_json_schema, + get_paper_title_system_prompt, + get_paper_title_validator_json_schema, + get_paper_title_validator_system_prompt, +) +from backend.autonomous.prompts.topic_exploration_prompts import ( + build_exploration_user_prompt, +) +from backend.autonomous.prompts.topic_prompts import ( + build_topic_selection_prompt, + build_topic_validation_prompt, + get_topic_selection_json_schema, + get_topic_selection_system_prompt, + get_topic_validator_json_schema, + get_topic_validator_system_prompt, +) + + +ENGINEERING_GOAL = ( + "Develop safer low-energy desalination for remote communities while reducing " + "membrane fouling, toxic cleaning chemicals, and lifecycle cost." +) +SCIENCE_GOAL = ( + "Develop a testable catalyst hypothesis for selective ambient-pressure ammonia " + "synthesis, including discriminating controls and measurable failure criteria." +) +SOFTWARE_GOAL = ( + "Design a resilient distributed protocol that preserves safety and useful " + "availability during partitions, Byzantine faults, and rolling upgrades." +) +MATH_GOAL = ( + "Prove a new combinatorial bound for intersecting set systems with a forbidden " + "configuration, including sharpness or a matching construction." +) +MIXED_GOAL = ( + "Optimize a flood-resilient microgrid under cost and repair constraints while " + "proving a formal impossibility bound on simultaneous outage and redundancy targets." +) +coordinator_module = importlib.import_module( + "backend.autonomous.core.autonomous_coordinator" +) + + +def _schema_fields(schema: str) -> set[str]: + return set(re.findall(r'^\s*"([a-z_]+)"\s*:', schema, flags=re.MULTILINE)) + + +def _assert_domain_general_strategy(text: str) -> None: + lower = text.lower() + assert "direct" in lower + assert "rigor" in lower + assert "math" in lower + assert "proof" in lower + assert any( + term in lower + for term in ("mechanism", "design", "algorithm", "experiment", "constraints") + ) + + +def _paper() -> dict: + return { + "paper_id": "paper_007", + "title": "Prior Membrane Study", + "reference_title_display": "Prior Membrane Study [validator ratings]", + "abstract": "A proposed cleaning mechanism with unresolved provenance.", + "outline": "I. Mechanism\nII. Constraints\nIII. Proposed tests", + "word_count": 1400, + "source_brainstorm_ids": ["topic_002"], + "content": "Full source content with proposed controls and limitations.", + } + + +@pytest.mark.parametrize( + ("module_name", "active_output"), + [ + ( + "topic_exploration_prompts", + build_exploration_user_prompt(ENGINEERING_GOAL, [], []), + ), + ("topic_prompts", get_topic_selection_system_prompt()), + ("completion_prompts", get_completion_review_system_prompt()), + ("paper_reference_prompts", get_pre_brainstorm_expansion_system_prompt(3)), + ( + "paper_title_exploration_prompts", + build_title_exploration_user_prompt( + ENGINEERING_GOAL, + "Reduce fouling safely.", + "A candidate cleaning mechanism.", + [], + [], + ), + ), + ("paper_title_prompts", get_paper_title_system_prompt()), + ("paper_continuation_prompts", get_continuation_decision_system_prompt()), + ], +) +def test_strategy_modules_do_not_impose_universal_mathematics_policy( + module_name: str, + active_output: str, +) -> None: + forbidden = re.compile( + r"\b(?:must be mathematical|must use mathematics|must include (?:a )?theorem|" + r"must include (?:a )?proof|only mathematical|mathematical form is required)\b", + flags=re.IGNORECASE, + ) + + assert not forbidden.search(active_output), module_name + assert re.search(r"\b(?:math|mathematical|theorem|proof)\w*\b", active_output, re.IGNORECASE) + assert re.search( + r"\b(?:when|whenever|where|if|conditional|appropriate|relevant)\b", + active_output, + re.IGNORECASE, + ) + + +def test_topic_exploration_preserves_full_engineering_goal_and_generalizes_routes() -> None: + prompt = build_exploration_user_prompt( + ENGINEERING_GOAL, + [{"topic_id": "topic_002", "topic_prompt": "Existing route", "status": "in_progress"}], + [_paper()], + ) + + assert ENGINEERING_GOAL in prompt + assert "strongest credible, genuinely novel" in prompt + assert "WHOLE question" in prompt + assert "easier adjacent" in prompt + for route in ("mechanism", "design route", "algorithm", "theorem", "experiment"): + assert route in prompt + assert "mathematics, theorem discovery, proof, and formalization remain first-class" in prompt + assert "evidence/provenance" in prompt + assert "proposed-test" in prompt + assert "mathematical direction" not in prompt.lower() + + +@pytest.mark.parametrize( + "system_prompt", + [ + get_topic_selection_system_prompt(), + get_topic_validator_system_prompt(), + ], +) +def test_topic_system_prompts_are_direct_novel_math_aware_and_non_fabricating( + system_prompt: str, +) -> None: + _assert_domain_general_strategy(system_prompt) + assert "genuinely novel" in system_prompt + assert "user's exact objective" in system_prompt + assert "whole question" in system_prompt.lower() + assert "easier" in system_prompt + assert "adjacent" in system_prompt or "background-heavy" in system_prompt + assert "provenance" in system_prompt or "evidence-aware" in system_prompt + assert "AI-GENERATED" in system_prompt + assert "NEVER cite internal documents as authoritative" in system_prompt + assert any( + phrase in system_prompt + for phrase in ( + "non-mathematical work is not deficient", + "Engineering, software, strategic, and causal routes", + "domain-grounded, evidence-aware", + ) + ) + + +def test_topic_builders_preserve_goal_candidates_feedback_and_phase_context() -> None: + brainstorms = [ + { + "topic_id": "topic_003", + "topic_prompt": "Partition recovery", + "status": "in_progress", + "submission_count": 8, + "papers_generated": ["paper_002"], + } + ] + action = { + "action": "continue_existing", + "topic_id": "topic_003", + "reasoning": "The recovery mechanism remains unresolved.", + } + selection = build_topic_selection_prompt( + SOFTWARE_GOAL, + brainstorms, + [_paper()], + rejection_context="Reject the broad survey detour.", + candidate_questions="Candidate 1: quorum repair under partitions", + ) + validation = build_topic_validation_prompt(SOFTWARE_GOAL, brainstorms, [_paper()], action) + + for prompt in (selection, validation): + assert SOFTWARE_GOAL in prompt + assert "topic_003" in prompt + assert "Prior Membrane Study" in prompt + assert "TOPIC EXPLORATION RESULTS" in selection + assert "Candidate 1: quorum repair under partitions" in selection + assert "Reject the broad survey detour." in selection + assert "PROPOSED TOPIC SELECTION" in validation + assert "continue_existing" in validation + assert "The recovery mechanism remains unresolved." in validation + + +def test_topic_schemas_preserve_exact_fields_and_enums() -> None: + selection = get_topic_selection_json_schema() + validator = get_topic_validator_json_schema() + + assert _schema_fields(selection) == {"action", "topic_id", "topic_prompt", "reasoning"} + assert '"new_topic", "continue_existing"' in selection + assert _schema_fields(validator) == {"decision", "reasoning"} + assert '"accept" or "reject"' in validator + + +@pytest.mark.asyncio +async def test_topic_execution_rejects_continuing_completed_brainstorm( + monkeypatch: pytest.MonkeyPatch, +) -> None: + coordinator = AutonomousCoordinator.__new__(AutonomousCoordinator) + coordinator._stop_event = SimpleNamespace(is_set=lambda: False) + coordinator._current_topic_id = "topic_before" + coordinator._acceptance_count = 4 + coordinator._consecutive_rejections = 2 + coordinator._broadcast = AsyncMock() + + metadata = SimpleNamespace( + status="complete", + submission_count=12, + papers_generated=["paper_001"], + ) + get_metadata = AsyncMock(return_value=metadata) + monkeypatch.setattr(coordinator_module.brainstorm_memory, "get_metadata", get_metadata) + + result = await coordinator._execute_topic_selection( + SimpleNamespace(action="continue_existing", topic_id="topic_complete") + ) + + assert result is None + assert coordinator._current_topic_id == "topic_before" + assert coordinator._acceptance_count == 4 + assert coordinator._consecutive_rejections == 2 + get_metadata.assert_awaited_once_with("topic_complete") + coordinator._broadcast.assert_awaited_once() + event_name, payload = coordinator._broadcast.await_args.args + assert event_name == "topic_selection_rejected" + assert "already marked complete" in payload["reasoning"] + + +@pytest.mark.parametrize( + "system_prompt", + [ + get_completion_review_system_prompt(), + get_completion_self_validation_system_prompt(), + ], +) +def test_completion_prompts_define_synthesis_readiness_without_empirical_overclaim( + system_prompt: str, +) -> None: + _assert_domain_general_strategy(system_prompt) + assert "AI-GENERATED" in system_prompt + assert "empirical or engineering work" in system_prompt + assert "synthesis" in system_prompt.lower() + assert "does NOT mean" in system_prompt or "does not assert" in system_prompt + assert any(term in system_prompt for term in ("experiments", "experiment")) + assert any(term in system_prompt for term in ("demonstrated", "demonstration")) + + +def test_completion_builders_preserve_science_goal_database_and_same_assessment_context() -> None: + review = build_completion_review_prompt( + SCIENCE_GOAL, + "Identify a falsifiable surface-poisoning mechanism.", + "Submission 1: proposed isotope control.", + 7, + "Prior review: add discriminating measurements.", + ) + assessment = { + "decision": "continue_brainstorm", + "reasoning": "Controls remain underspecified.", + "suggested_additions": "Specify isotope and blank controls.", + } + self_validation = build_completion_self_validation_prompt( + SCIENCE_GOAL, + "Identify a falsifiable surface-poisoning mechanism.", + "Submission 1: proposed isotope control.", + assessment, + ) + + for prompt in (review, self_validation): + assert SCIENCE_GOAL in prompt + assert "Identify a falsifiable surface-poisoning mechanism." in prompt + assert "Submission 1: proposed isotope control." in prompt + assert "Total Accepted Submissions: 7" in review + assert "Prior review: add discriminating measurements." in review + assert "YOUR COMPLETION ASSESSMENT (to validate)" in self_validation + assert "continue_brainstorm" in self_validation + assert "Specify isotope and blank controls." in self_validation + + +def test_completion_schemas_preserve_exact_fields_and_enums() -> None: + review = get_completion_review_json_schema() + self_validation = get_completion_self_validation_json_schema() + + assert _schema_fields(review) == {"decision", "reasoning", "suggested_additions"} + assert '"continue_brainstorm" or "write_paper"' in review + assert _schema_fields(self_validation) == {"validated", "reasoning"} + assert "true or false" in self_validation + + +@pytest.mark.parametrize( + ("system_prompt", "cap_text"), + [ + (get_pre_brainstorm_expansion_system_prompt(3), "up to 3 papers maximum"), + (get_additional_reference_expansion_system_prompt(3), "max 3 total"), + (get_reference_expansion_system_prompt(6), "up to 6 papers"), + (get_reference_selection_system_prompt(4), "Maximum 4 papers"), + ], +) +def test_reference_system_prompts_preserve_caps_math_provenance_and_skepticism( + system_prompt: str, cap_text: str +) -> None: + assert cap_text in system_prompt + assert "AI-GENERATED" in system_prompt + assert "NEVER cite internal documents as authoritative" in system_prompt + assert "Mathematical reasoning" in system_prompt + assert "proof" in system_prompt.lower() + assert "mechanism" in system_prompt.lower() + assert "evidence" in system_prompt.lower() + assert ( + "provenance" in system_prompt.lower() + or "independently re-checking" in system_prompt + or "verify information independently" in system_prompt + ) + assert "direct" in system_prompt.lower() + + +def test_reference_builders_preserve_goal_two_step_context_persistence_and_rendered_caps() -> None: + paper = _paper() + pre = build_pre_brainstorm_expansion_prompt( + ENGINEERING_GOAL, "Reduce fouling safely.", "[Not started]", [paper], 3 + ) + additional = build_additional_reference_expansion_prompt( + ENGINEERING_GOAL, + "Reduce fouling safely.", + "A candidate pulsed-cleaning mechanism.", + [paper], + ["paper_001"], + [{"paper_id": "paper_001", "title": "Already selected"}], + 3, + ) + legacy = build_reference_expansion_prompt( + ENGINEERING_GOAL, "Reduce fouling safely.", "Brainstorm summary", [paper], 6 + ) + final = build_reference_selection_prompt( + ENGINEERING_GOAL, + "Reduce fouling safely.", + "Brainstorm summary", + [paper], + mode="initial", + max_papers=3, + retrieved_context="Retrieved evidence with source provenance.", + ) + + for prompt in (pre, additional, legacy, final): + assert ENGINEERING_GOAL in prompt + assert "Reduce fouling safely." in prompt + assert "paper_007" in prompt + assert "Prior Membrane Study [validator ratings]" in prompt + assert "ENTIRE brainstorm exploration AND paper writing" in pre + assert "abstracts and outlines" in pre + assert "1 papers, 2 slots remaining" in additional + assert "can add up to 2 more" in additional + assert "Already selected" in additional + assert "Titles, Abstracts, and Outlines" in legacy + assert "MODE: INITIAL SELECTION" in final + assert "FULL PAPER CONTENT" in final + assert "RAG-RETRIEVED FULL-PAPER EVIDENCE" in final + assert "up to 3 papers maximum" in final + assert "brainstorm exploration AND paper writing" in final + + +def test_reference_schemas_preserve_exact_fields_and_caps() -> None: + expansion = get_reference_expansion_json_schema() + selection = get_reference_selection_json_schema(3) + + assert _schema_fields(expansion) == { + "expand_papers", + "proceed_without_references", + "reasoning", + } + assert _schema_fields(selection) == {"selected_papers", "reasoning"} + assert "maximum 3" in selection + assert "up to 3 paper_ids" in selection + + +def test_title_exploration_preserves_goal_and_uses_five_domain_appropriate_candidates() -> None: + prompt = build_title_exploration_user_prompt( + MATH_GOAL, + "Find the sharp extremal obstruction.", + "A compression argument suggests a bound.", + [_paper()], + [_paper()], + ) + + assert MATH_GOAL in prompt + assert "collect 5 validated candidate titles" in prompt + assert "solution-oriented research paper or report" in prompt + assert "mathematical title forms when the work is mathematical" in prompt + assert "theorems, and proofs remain first-class" in prompt + assert "does not imply" in prompt.lower() + assert "completed experiments" in prompt + assert "EXISTING RELATED PAPERS (do not duplicate" in prompt + + +@pytest.mark.parametrize( + "system_prompt", + [ + get_paper_title_system_prompt(), + get_paper_title_validator_system_prompt(), + ], +) +def test_title_system_prompts_are_domain_general_math_aware_and_anti_fabrication( + system_prompt: str, +) -> None: + assert "paper" in system_prompt.lower() + assert "domain" in system_prompt.lower() + assert "Mathematical reasoning" in system_prompt + assert "first-class whenever relevant" in system_prompt + assert "AI-GENERATED" in system_prompt + assert "provenance" in system_prompt + assert any( + phrase in system_prompt.lower() + for phrase in ( + "proposed experiments", + "completed empirical demonstration", + "implemented, tested, measured, or demonstrated", + ) + ) + + +def test_title_builders_preserve_mixed_goal_candidates_feedback_and_redundancy_boundary() -> None: + existing = [{"title": "Completed Reliability Bound", "abstract": "A finished prior result."}] + selection = build_paper_title_prompt( + MIXED_GOAL, + "Joint engineering and impossibility analysis.", + "Source brainstorm: optimize topology and prove a lower bound.", + existing, + [_paper()], + rejection_feedback="Do not imply the pilot was completed.", + candidate_titles="1. Cost-Constrained Recovery with an Impossibility Frontier", + ) + validation = build_paper_title_validation_prompt( + MIXED_GOAL, + "Joint engineering and impossibility analysis.", + "Source brainstorm: optimize topology and prove a lower bound.", + existing, + "A Proposed Microgrid Design and Formal Outage Bound", + "Reflects the source brainstorm without duplicating the completed paper.", + [_paper()], + ) + + for prompt in (selection, validation): + assert MIXED_GOAL in prompt + assert "Source brainstorm: optimize topology and prove a lower bound." in prompt + assert "Completed Reliability Bound" in prompt + assert "SELECTED REFERENCE PAPERS" in prompt + assert "PRE-VALIDATED CANDIDATE TITLES" in selection + assert "Do not imply the pilot was completed." in selection + assert "EXISTING PAPERS FROM THIS BRAINSTORM (Differentiate from these)" in selection + assert "PROPOSED TITLE" in validation + validator = get_paper_title_validator_system_prompt() + assert "DO NOT reject a title for being \"similar to brainstorm submissions\"" in validator + assert "ONLY reject for similarity" in validator + assert "EXISTING COMPLETED PAPERS" in validator + + +def test_title_schemas_preserve_exact_fields_and_enums() -> None: + title = get_paper_title_json_schema() + validator = get_paper_title_validator_json_schema() + + assert _schema_fields(title) == {"paper_title", "reasoning"} + assert _schema_fields(validator) == {"decision", "reasoning"} + assert '"accept" or "reject"' in validator + + +@pytest.mark.parametrize( + "system_prompt", + [ + get_continuation_decision_system_prompt(), + get_continuation_validator_system_prompt(), + ], +) +def test_continuation_system_prompts_support_distinct_answer_bearing_contributions( + system_prompt: str, +) -> None: + assert "direct answer" in system_prompt + assert "domain- and claim-appropriate rigor" in system_prompt + assert "proof" in system_prompt.lower() + assert "mechanism" in system_prompt.lower() + assert "evidence" in system_prompt.lower() + assert "AI-GENERATED" in system_prompt + assert "provenance" in system_prompt or "without pretending" in system_prompt + + +def test_continuation_builders_preserve_goal_phase_context_decision_and_three_paper_cap() -> None: + papers = [ + { + "title": "Protocol Core", + "abstract": "Safety mechanism and assumptions.", + "outline": "I. Model\nII. Protocol", + } + ] + decision = build_continuation_decision_prompt( + SOFTWARE_GOAL, + "Partition-tolerant protocol.", + "Submissions include a distinct recovery design.", + papers, + 2, + "Previous feedback: distinguish the recovery contribution.", + ) + validation = build_continuation_validation_prompt( + SOFTWARE_GOAL, + "Partition-tolerant protocol.", + "Submissions include a distinct recovery design.", + papers, + 2, + { + "decision": "write_another_paper", + "reasoning": "Recovery is a distinct answer-bearing contribution.", + }, + ) + + for prompt in (decision, validation): + assert SOFTWARE_GOAL in prompt + assert "Partition-tolerant protocol." in prompt + assert "Submissions include a distinct recovery design." in prompt + assert "Protocol Core" in prompt + assert "2 of 3 maximum" in prompt + assert "Previous feedback: distinguish the recovery contribution." in decision + assert "PROPOSED CONTINUATION DECISION" in validation + assert "write_another_paper" in validation + + +def test_continuation_schemas_preserve_exact_fields_and_enums() -> None: + decision = get_continuation_decision_json_schema() + validator = get_continuation_validator_json_schema() + + assert _schema_fields(decision) == {"decision", "reasoning"} + assert '"write_another_paper" or "move_on"' in decision + assert _schema_fields(validator) == {"decision", "reasoning"} + assert '"accept" or "reject"' in validator diff --git a/tests/unit/test_all_purpose_compiler_prompts.py b/tests/unit/test_all_purpose_compiler_prompts.py new file mode 100644 index 0000000..c12bdb8 --- /dev/null +++ b/tests/unit/test_all_purpose_compiler_prompts.py @@ -0,0 +1,535 @@ +from pathlib import Path +import json +from unittest.mock import AsyncMock + +import pytest + +from backend.compiler.memory.compiler_rejection_log import CompilerRejectionLog +from backend.compiler.memory.paper_memory import ( + PAPER_ANCHOR, + THEOREMS_APPENDIX_END, + THEOREMS_APPENDIX_START, +) +from backend.compiler.prompts.construction_prompts import ( + get_abstract_construction_system_prompt, + get_body_construction_system_prompt, + get_conclusion_construction_system_prompt, + get_construction_json_schema, + get_construction_system_prompt, + get_introduction_construction_system_prompt, + get_wolfram_tool_guidance, +) +from backend.compiler.prompts.critique_prompts import ( + get_critique_json_schema, + get_critique_submitter_system_prompt, + get_critique_validation_json_schema, + get_critique_validator_system_prompt, +) +from backend.compiler.prompts.outline_prompts import ( + build_outline_create_prompt, + build_outline_update_prompt, + get_outline_create_system_prompt, + get_outline_json_schema, + get_outline_update_system_prompt, +) +from backend.compiler.prompts.review_prompts import ( + EMPIRICAL_RED_TEAM_REVIEW_FOCUS, + get_review_json_schema, + get_review_system_prompt, +) +from backend.compiler.validation.compiler_validator import CompilerValidator +from backend.compiler.core.compiler_coordinator import CompilerCoordinator +from backend.compiler.agents.writer_submitter import ( + WOLFRAM_MAX_CALLS_PER_SUBMISSION, + WritingSubmitter, +) +from backend.shared.critique_prompts import ( + CRITIQUE_JSON_SCHEMA, + build_critique_prompt as build_shared_critique_prompt, + get_default_critique_prompt, +) +from backend.shared.models import CompilerValidationResult, ContextPack + + +def _joined(*parts: str) -> str: + return "\n".join(parts).lower() + + +def _make_writer(monkeypatch) -> WritingSubmitter: + from backend.compiler.agents import writer_submitter + + monkeypatch.setattr( + writer_submitter.system_config, + "compiler_writer_context_window", + 32768, + ) + monkeypatch.setattr( + writer_submitter.system_config, + "compiler_writer_max_output_tokens", + 4096, + ) + return WritingSubmitter("test-model", "Solve it.") + + +def test_outline_contract_supports_domain_specific_solution_structures() -> None: + prompt = _joined( + get_outline_create_system_prompt(), + get_outline_update_system_prompt(), + ) + + for concept in ( + "algorithm", + "architecture", + "engineering", + "failure modes", + "empirical", + "falsifiable", + "strateg", + "evaluation", + ): + assert concept in prompt + assert "direct solution" in prompt or "direct answer" in prompt + assert "non-mathematical work must not be forced into mathematical form" in prompt + + +def test_all_construction_phase_prompts_allow_empty_new_string_only_for_delete() -> None: + for prompt in ( + get_construction_system_prompt(), + get_body_construction_system_prompt(), + get_conclusion_construction_system_prompt(), + get_introduction_construction_system_prompt(), + get_abstract_construction_system_prompt(), + get_construction_json_schema(), + ): + lowered = prompt.lower() + if "new_string" in lowered and "must" in lowered and "empty" in lowered: + assert "delete" in lowered + + +@pytest.mark.parametrize( + ("outline", "expected_error"), + [ + ( + "I. Abstract\nII. Introduction\nIII. Method\nIV. Conclusion", + "INCORRECT_SECTION_NAME - Abstract", + ), + ( + "I. Introduction\nAbstract\nII. Method\nIII. Conclusion", + "INCORRECT_SECTION_ORDER - Abstract", + ), + ( + "I. Introduction\nII. Conclusion", + "MISSING_REQUIRED_SECTION - Body", + ), + ], +) +def test_outline_prevalidation_enforces_abstract_and_body_structure( + outline: str, + expected_error: str, +) -> None: + coordinator = object.__new__(CompilerCoordinator) + assert expected_error in coordinator._pre_validate_outline_structure(outline) + + +def test_outline_contract_retains_mathematical_progression_conditionally() -> None: + prompt = _joined( + get_outline_create_system_prompt(), + get_outline_update_system_prompt(), + ) + + for concept in ("definitions", "theorems", "proofs"): + assert concept in prompt + assert "mathemat" in prompt + assert "when relevant" in prompt or "when useful" in prompt or "as appropriate" in prompt + + +def test_outline_contract_allows_only_unnumbered_optional_abstract() -> None: + prompt = _joined( + get_outline_create_system_prompt(), + get_outline_update_system_prompt(), + get_outline_json_schema(), + ) + + assert 'unnumbered heading "abstract"' in prompt + assert "i. introduction" in prompt + assert "i. abstract" not in prompt + assert "0. abstract" not in prompt + + +@pytest.mark.asyncio +async def test_active_outline_builders_preserve_abstract_and_update_contracts( + monkeypatch, +) -> None: + from backend.compiler.prompts import outline_prompts + + monkeypatch.setattr( + outline_prompts.compiler_rejection_log, + "get_rejections_text", + AsyncMock(return_value=""), + ) + monkeypatch.setattr( + "backend.compiler.memory.outline_memory.outline_memory.get_creation_feedback", + AsyncMock(return_value=""), + ) + + create_prompt = await build_outline_create_prompt("Solve it.", "Evidence") + update_prompt = await build_outline_update_prompt( + "Solve it.", + "I. Introduction\nII. Body\nIII. Conclusion", + "Draft", + ) + + assert 'unnumbered heading "Abstract"' in create_prompt + assert "I. Introduction" in create_prompt + assert "I. Abstract" not in create_prompt + assert "0. Abstract" not in create_prompt + assert '"operation": "insert_after"' in update_prompt + assert "insert_after | replace" not in update_prompt + + +@pytest.mark.asyncio +async def test_submit_outline_update_rejects_affirmative_non_insert_operation( + monkeypatch, +) -> None: + from backend.compiler.agents import writer_submitter + + submitter = _make_writer(monkeypatch) + monkeypatch.setattr(writer_submitter.outline_memory, "get_outline", AsyncMock(return_value="I. Introduction")) + monkeypatch.setattr(writer_submitter.paper_memory, "get_paper", AsyncMock(return_value="Draft")) + monkeypatch.setattr( + writer_submitter.compiler_rag_manager, + "retrieve_for_mode", + AsyncMock(return_value=ContextPack(text="")), + ) + monkeypatch.setattr(writer_submitter, "build_outline_update_prompt", AsyncMock(return_value="prompt")) + monkeypatch.setattr(writer_submitter, "count_tokens", lambda _text: 1) + monkeypatch.setattr( + writer_submitter.api_client_manager, + "prewarm_assistant_memory_context", + AsyncMock(return_value=None), + ) + monkeypatch.setattr( + writer_submitter.api_client_manager, + "generate_completion", + AsyncMock( + return_value={ + "choices": [{ + "message": { + "content": json.dumps({ + "needs_update": True, + "operation": "replace", + "old_string": "I. Introduction", + "new_string": "I. Revised Introduction", + "reasoning": "Revise.", + }) + } + }] + } + ), + ) + + with pytest.raises(ValueError, match="operation='insert_after'"): + await submitter.submit_outline_update() + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("operation", "old_string", "new_string", "expected_content", "accepted"), + [ + ("delete", "obsolete paragraph", "", "obsolete paragraph", True), + ("delete", "", "", None, False), + ("replace", "old", "new", "new", True), + ("insert_after", "anchor", "addition", "addition", True), + ("full_content", "", "complete draft", "complete draft", True), + ("replace", "old", "", None, False), + ("insert_after", "anchor", "", None, False), + ("full_content", "", "", None, False), + ], +) +async def test_construction_parser_validates_fields_by_operation( + monkeypatch, + operation, + old_string, + new_string, + expected_content, + accepted, +) -> None: + from backend.compiler.agents import writer_submitter + + submitter = _make_writer(monkeypatch) + payload = { + "needs_construction": True, + "operation": operation, + "old_string": old_string, + "new_string": new_string, + "reasoning": "Edit the draft.", + "section_complete": False, + } + monkeypatch.setattr(writer_submitter.outline_memory, "get_outline", AsyncMock(return_value="I. Introduction")) + monkeypatch.setattr(writer_submitter.paper_memory, "get_paper", AsyncMock(return_value="Draft")) + monkeypatch.setattr( + writer_submitter.compiler_rag_manager, + "retrieve_for_mode", + AsyncMock(return_value=ContextPack(text="")), + ) + monkeypatch.setattr(writer_submitter, "build_construction_prompt", AsyncMock(return_value="prompt")) + monkeypatch.setattr(writer_submitter, "count_tokens", lambda _text: 1) + monkeypatch.setattr( + writer_submitter.api_client_manager, + "prewarm_assistant_memory_context", + AsyncMock(return_value=None), + ) + monkeypatch.setattr( + submitter, + "_generate_completion_with_wolfram_tool", + AsyncMock(return_value=(json.dumps(payload), [], {})), + ) + + submission = await submitter.submit_construction() + + assert (submission is not None) is accepted + if submission is not None: + assert submission.operation == operation + assert submission.old_string == old_string + assert submission.new_string == new_string + assert submission.content == expected_content + + +def test_body_construction_supports_all_purpose_solution_content() -> None: + prompt = get_body_construction_system_prompt().lower() + + for concept in ( + "algorithm", + "engineering", + "empirical", + "strateg", + "constraints", + "failure modes", + ): + assert concept in prompt + assert "independently checkable" in prompt + assert "section_complete" in prompt + + +def test_construction_retains_mathematical_rigor_and_provenance_guards() -> None: + prompt = _joined( + get_body_construction_system_prompt(), + get_construction_system_prompt(), + ) + + assert "mathematical claims" in prompt + assert "proof" in prompt + assert "explicit assumptions" in prompt + for prohibited in ( + "invent citations", + "experiments", + "benchmark numbers", + "code artifacts", + ): + assert prohibited in prompt + + +def test_review_checks_direct_value_and_domain_specific_failure_modes() -> None: + prompt = get_review_system_prompt().lower() + + assert "direct" in prompt + assert "user's" in prompt and "prompt" in prompt + assert "failure modes" in prompt + assert "interfaces" in prompt or "implementation" in prompt + assert "mathematical" in prompt and ("proof gap" in prompt or "proof gaps" in prompt) + assert "no edit" in prompt or "needs_edit" in prompt + + +def test_empirical_red_team_remains_conservative_and_non_fabricating() -> None: + prompt = EMPIRICAL_RED_TEAM_REVIEW_FOCUS.lower() + + for concept in ( + "fabricated experiments", + "nonexistent code", + "unsupported benchmark", + "uncited external", + "proposed experiment", + ): + assert concept in prompt + assert "do not preserve unsupported benchmark numbers" in prompt + + +def test_compiler_self_review_accepts_engineering_and_algorithmic_limitations() -> None: + submitter = get_critique_submitter_system_prompt().lower() + validator = get_critique_validator_system_prompt().lower() + + for concept in ("engineering", "algorithm", "implementation"): + assert concept in submitter or concept in validator + assert "one important point per turn" in submitter + assert "do not propose direct edits or rewrites" in submitter + assert "critique_needed=false" in submitter + assert "non-redundant" in validator + + +def test_ordinary_validator_uses_domain_rigor_and_keeps_theorem_guard() -> None: + validator = CompilerValidator(model_name="test-model", user_prompt="Solve it.") + outline = validator._get_outline_validation_system_prompt("outline_create").lower() + construction = validator._get_paper_validation_system_prompt("construction").lower() + + for concept in ( + "domain-appropriate", + "direct solution value", + "claim-type provenance", + "specificity and actionability", + "novelty without fabrication", + ): + assert concept in outline or concept in construction + assert "does not require mathematics" in construction or "mathematics is irrelevant" in construction + assert "unsupported theorem claims" in construction + assert "rigor_check" in construction + + +def test_prompt_schemas_preserve_existing_field_contracts() -> None: + outline_schema = get_outline_json_schema() + construction_schema = get_construction_json_schema() + review_schema = get_review_json_schema() + critique_schema = get_critique_json_schema() + critique_validation_schema = get_critique_validation_json_schema() + + for field in ("content", "outline_complete", "reasoning"): + assert field in outline_schema + for field in ( + "needs_construction", + "operation", + "old_string", + "new_string", + "reasoning", + "section_complete", + ): + assert field in construction_schema + for field in ("needs_edit", "operation", "old_string", "new_string", "reasoning"): + assert field in review_schema + for field in ("critique_needed", "submission", "reasoning"): + assert field in critique_schema + for field in ("decision", "reasoning", "summary"): + assert field in critique_validation_schema + + +def test_exact_edit_and_protected_markers_remain_visible() -> None: + prompts = ( + get_body_construction_system_prompt(), + get_conclusion_construction_system_prompt(), + get_introduction_construction_system_prompt(), + get_abstract_construction_system_prompt(), + get_construction_system_prompt(), + get_review_system_prompt(), + ) + + assert THEOREMS_APPENDIX_START == ( + "[HARD CODED THEOREMS APPENDIX START -- LEAN 4 VERIFIED THEOREMS BELOW]" + ) + assert THEOREMS_APPENDIX_END == ( + "[HARD CODED THEOREMS APPENDIX END -- ALL APPENDIX CONTENT SHOULD BE ABOVE THIS LINE]" + ) + assert PAPER_ANCHOR == ( + "[HARD CODED END-OF-PAPER MARK -- ALL CONTENT SHOULD BE ABOVE THIS LINE]" + ) + for prompt in prompts: + assert "old_string" in prompt + assert "new_string" in prompt + marker_prompts = "\n".join(prompts) + assert PAPER_ANCHOR in marker_prompts + assert THEOREMS_APPENDIX_START in marker_prompts + assert THEOREMS_APPENDIX_END in marker_prompts + + +def test_ordinary_prompt_identities_are_not_universally_mathematical() -> None: + prompts = ( + get_outline_create_system_prompt(), + get_outline_update_system_prompt(), + get_body_construction_system_prompt(), + get_conclusion_construction_system_prompt(), + get_introduction_construction_system_prompt(), + get_abstract_construction_system_prompt(), + get_construction_system_prompt(), + get_review_system_prompt(), + get_critique_submitter_system_prompt(), + get_critique_validator_system_prompt(), + ) + + for prompt in prompts: + assert "mathematical document" not in prompt.lower() + + +def test_shared_default_critique_is_domain_general_and_schema_stable() -> None: + prompt = get_default_critique_prompt().lower() + + for standard in ("factual", "logical", "mathematical", "technical", "methodological"): + assert standard in prompt + for field in ( + "novelty_rating", + "novelty_feedback", + "correctness_rating", + "correctness_feedback", + "impact_rating", + "impact_feedback", + "full_critique", + ): + assert field in CRITIQUE_JSON_SCHEMA + + +def test_shared_custom_critique_prompt_remains_authoritative() -> None: + custom = "CUSTOM REVIEW AUTHORITY: assess only the supplied deployment criteria." + built = build_shared_critique_prompt( + paper_content="Paper body.", + paper_title="Deployment report", + custom_prompt=custom, + ) + + assert custom in built + assert get_default_critique_prompt() not in built + assert CRITIQUE_JSON_SCHEMA in built + assert "PAPER TITLE: Deployment report" in built + assert "Paper body." in built + + +def test_wolfram_guidance_preserves_tool_and_budget_contract(monkeypatch) -> None: + from backend.compiler.prompts import construction_prompts + from backend.shared import wolfram_alpha_client + + monkeypatch.setattr(construction_prompts.system_config, "wolfram_alpha_enabled", True) + monkeypatch.setattr(construction_prompts.system_config, "wolfram_alpha_api_key", "test-key") + monkeypatch.setattr(wolfram_alpha_client, "get_wolfram_client", lambda: object()) + + guidance = get_wolfram_tool_guidance() + + assert "CONSTRUCTION MODE ONLY" in guidance + assert "wolfram_alpha_query" in guidance + assert "up to 20 Wolfram Alpha calls" in guidance + assert "Lean 4 proof verification" in guidance + assert WOLFRAM_MAX_CALLS_PER_SUBMISSION == 20 + + +@pytest.mark.asyncio +async def test_rejection_repair_guidance_is_domain_appropriate(tmp_path: Path) -> None: + log = CompilerRejectionLog() + log.rejections_file = tmp_path / "rejections.txt" + log.acceptances_file = tmp_path / "acceptances.txt" + log.declines_file = tmp_path / "declines.txt" + await log.initialize() + + await log.add_rejection( + CompilerValidationResult( + submission_id="domain-rigor", + decision="reject", + reasoning="The mechanism has no feasibility support.", + summary="Correctness standard failed.", + coherence_check=True, + rigor_check=False, + placement_check=True, + ), + mode="construction", + submission_content="Proposed mechanism.", + ) + text = await log.get_rejections_text() + + assert "appropriate to their domain and claim type" in text + assert "Mathematical claims require sound derivation, proof, or explicit assumptions" in text + assert "factual, logical, technical, empirical, and methodological claims" in text + assert "SUBMISSION PREVIEW" in text + assert "VALIDATOR REASONING" in text diff --git a/tests/unit/test_all_purpose_prompt_contracts.py b/tests/unit/test_all_purpose_prompt_contracts.py new file mode 100644 index 0000000..8caba18 --- /dev/null +++ b/tests/unit/test_all_purpose_prompt_contracts.py @@ -0,0 +1,289 @@ +from types import SimpleNamespace + +import pytest + +from backend.aggregator.prompts.submitter_prompts import ( + build_submitter_prompt, + get_submitter_system_prompt, +) +from backend.aggregator.prompts.validator_prompts import get_validator_system_prompt +from backend.autonomous.core.autonomous_coordinator import AutonomousCoordinator +from backend.autonomous.prompts.completion_prompts import ( + build_completion_review_prompt, + get_completion_review_system_prompt, +) +from backend.autonomous.prompts.final_answer_prompts import ( + build_certainty_assessment_prompt, + get_certainty_assessment_system_prompt, + get_format_selection_system_prompt, + get_volume_organization_system_prompt, +) +from backend.autonomous.prompts.paper_redundancy_prompts import ( + build_paper_redundancy_prompt, +) +from backend.autonomous.prompts.paper_title_prompts import build_paper_title_prompt +from backend.autonomous.prompts.proof_prompts import PROOF_FRAMING_CONTEXT +from backend.autonomous.prompts.topic_exploration_prompts import ( + build_exploration_user_prompt, +) +from backend.autonomous.prompts.topic_prompts import build_topic_selection_prompt +from backend.compiler.prompts.construction_prompts import ( + get_body_construction_system_prompt, +) +from backend.compiler.prompts.outline_prompts import ( + build_outline_create_prompt, + get_outline_create_system_prompt, +) +from backend.compiler.validation.compiler_validator import CompilerValidator +from backend.shared.api_client_manager import APIClientManager +from backend.shared.models import BrainstormRetroactiveOperation + + +OBJECTIVES = ( + "Prove a new additive-combinatorics bound and characterize sharpness.", + "Design safer low-energy desalination under fouling and lifecycle constraints.", + "Create a resilient distributed protocol for Byzantine faults and rolling upgrades.", + "Propose a catalyst hypothesis with discriminating controls and a test plan.", + "Optimize a physical design subject to a formally provable safety bound.", + "Determine whether physical and budget constraints make the exact objective impossible.", +) + +PAPERS = [ + { + "paper_id": "paper_001", + "title": "Candidate mechanism", + "abstract": "A proposed mechanism with simulation evidence and unperformed tests.", + "outline": "Mechanism; constraints; validation", + "word_count": 1200, + } +] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("objective", OBJECTIVES) +async def test_exact_objective_survives_representative_active_builders( + objective: str, +) -> None: + prompts = [ + build_submitter_prompt(objective, "Current accepted evidence."), + build_exploration_user_prompt(objective, [], []), + build_topic_selection_prompt(objective, [], [], ["Candidate route"]), + build_completion_review_prompt( + objective, + "Current topic", + "Accepted contribution", + 7, + "", + ), + build_paper_title_prompt( + objective, + "Current topic", + "Accepted contribution", + [], + [], + ["Candidate title"], + ), + await build_outline_create_prompt(objective, "Optional source evidence."), + build_certainty_assessment_prompt(objective, PAPERS), + build_paper_redundancy_prompt(objective, PAPERS), + ] + + for prompt in prompts: + assert objective in prompt + + +def test_cross_build_policy_is_domain_general_and_proof_positive() -> None: + ordinary_policy = "\n".join( + ( + get_submitter_system_prompt(), + get_validator_system_prompt(), + get_completion_review_system_prompt(), + get_outline_create_system_prompt(), + get_body_construction_system_prompt(), + CompilerValidator._get_paper_validation_system_prompt( + object.__new__(CompilerValidator), + "construction", + ), + get_certainty_assessment_system_prompt(), + get_format_selection_system_prompt(), + get_volume_organization_system_prompt(), + ) + ).lower() + + for required in ( + "exact objective", + "novel", + "domain", + "claim", + "mathemat", + "proof", + "evidence", + "provenance", + ): + assert required in ordinary_policy + assert "non-mathematical work must not be rejected" in ordinary_policy + assert "when relevant" in ordinary_policy + + +def test_active_ordinary_paths_have_no_universal_mathematics_gate() -> None: + validator = object.__new__(CompilerValidator) + operations = ( + BrainstormRetroactiveOperation( + action="delete", + submission_number=1, + reasoning="The claimed benchmark was never run.", + ), + BrainstormRetroactiveOperation( + action="edit", + submission_number=1, + new_content="Treat the benchmark as a proposed validation plan.", + reasoning="Restore the evidence status.", + ), + BrainstormRetroactiveOperation( + action="add", + new_content="Add a concrete fault-injection protocol and acceptance criteria.", + reasoning="This closes an implementation-validation gap.", + ), + ) + ordinary_prompts = [ + get_submitter_system_prompt(), + get_validator_system_prompt(), + get_completion_review_system_prompt(), + get_outline_create_system_prompt(), + get_body_construction_system_prompt(), + get_certainty_assessment_system_prompt(), + get_format_selection_system_prompt(), + get_volume_organization_system_prompt(), + *[ + validator._build_brainstorm_validation_prompt(op, "Existing database") + for op in operations + ], + ] + forbidden = ( + "you are a mathematical submitter", + "user's mathematical prompt", + "all claims must be grounded in established mathematical principles", + "must include a theorem", + "must include a proof", + ) + + for prompt in ordinary_prompts: + lowered = prompt.lower() + for phrase in forbidden: + assert phrase not in lowered + + retroactive = "\n".join(ordinary_prompts[-3:]).lower() + assert "domain and claim types" in retroactive + assert "mechanism, design, algorithm, experiment" in retroactive + assert "protected lean 4 proofs" in retroactive + + +def test_empirical_status_is_not_promoted_across_builds() -> None: + aggregator = get_submitter_system_prompt().lower() + completion = get_completion_review_system_prompt().lower() + compiler = get_body_construction_system_prompt().lower() + certainty = get_certainty_assessment_system_prompt().lower() + + assert "proposed experiment" in aggregator + assert "does not mean a proposed mechanism has been built" in completion + assert "never invent citations, experiments" in compiler + for status in ("proposals", "hypotheses", "required validation", "merely because it is proposed"): + assert status in certainty + + +def test_assistant_memory_stays_optional_and_formal_context_is_targeted() -> None: + engineering_prompt = ( + "USER PROMPT:\nDesign safer low-energy desalination.\n\n" + "YOUR TASK:\nPropose a concrete mechanism and validation plan." + ) + engineering = APIClientManager._build_assistant_target_snapshot( + "aggregator_submitter_1", + "agg_sub1_001", + engineering_prompt, + ) + formal = APIClientManager._build_assistant_target_snapshot( + "autonomous_proof_formalizer", + "proof_form_001", + "USER PROMPT:\nProve the target bound.\n\nTARGET THEOREM:\nBoundedInvariant", + ) + injected = APIClientManager._append_assistant_memory_block( + engineering_prompt, + "Lean-verified support record", + ).lower() + + assert engineering.imports == [] + assert formal.imports == ["Mathlib"] + assert "optional" in injected + assert "cannot redirect or mathematically reinterpret" in injected + assert "do not require mathematics or formal proof" in injected + + +@pytest.mark.asyncio +async def test_bounded_nonmathematical_papers_only_assembly_has_no_theorem_gate() -> None: + objective = OBJECTIVES[1] + chain = "\n".join( + ( + build_exploration_user_prompt(objective, [], []), + build_topic_selection_prompt(objective, [], [], ["Membrane route"]), + build_submitter_prompt(objective, "A proposed cleaning mechanism."), + build_completion_review_prompt( + objective, + "Membrane route", + "A proposed cleaning mechanism.", + 7, + "", + ), + build_paper_title_prompt( + objective, + "Membrane route", + "A proposed cleaning mechanism.", + [], + [], + ["Safer Low-Energy Desalination"], + ), + await build_outline_create_prompt(objective, "A proposed cleaning mechanism."), + get_body_construction_system_prompt(), + ) + ).lower() + + assert objective.lower() in chain + assert "mechanism" in chain + assert "verification" in chain + assert "non-mathematical work must not be forced into mathematical form" in chain + assert "must include a theorem" not in chain + assert "must include a proof" not in chain + + +def test_mixed_and_mathematical_paths_keep_proof_boundaries_available() -> None: + mixed = OBJECTIVES[4] + coordinator = object.__new__(AutonomousCoordinator) + coordinator._proof_framing_active = True + coordinator._proof_framing_context = PROOF_FRAMING_CONTEXT + framed = coordinator._append_proof_framing( + build_submitter_prompt(mixed, "Engineering design evidence.") + ) + rigor = CompilerValidator._get_mathematical_rigor_validation_system_prompt( + object.__new__(CompilerValidator), + "rigor", + ).lower() + + assert mixed in framed + assert "PROOF_FRAMING_CONTEXT" not in framed + assert "directly help answer, support, or advance" in framed + assert "mathematical rigor" in rigor + assert "proof" in rigor + + +def test_manual_aggregator_compiler_contract_remains_domain_appropriate() -> None: + objective = OBJECTIVES[2] + aggregator = build_submitter_prompt(objective, "Existing protocol idea.") + compiler = get_body_construction_system_prompt() + validator = CompilerValidator._get_paper_validation_system_prompt( + object.__new__(CompilerValidator), + "construction", + ) + + assert objective in aggregator + assert "algorithm" in aggregator.lower() + assert "domain" in compiler.lower() + assert "non-mathematical work must not be rejected" in validator.lower() diff --git a/tests/unit/test_all_purpose_tier3_prompts.py b/tests/unit/test_all_purpose_tier3_prompts.py new file mode 100644 index 0000000..77d7b19 --- /dev/null +++ b/tests/unit/test_all_purpose_tier3_prompts.py @@ -0,0 +1,581 @@ +from types import SimpleNamespace + +import pytest +import importlib +from backend.autonomous.core.autonomous_coordinator import AutonomousCoordinator +from backend.autonomous.agents.final_answer.certainty_assessor import CertaintyAssessor +from backend.autonomous.agents.final_answer.volume_organizer import VolumeOrganizer +from backend.autonomous.prompts.final_answer_prompts import ( + build_certainty_assessment_prompt, + build_certainty_validation_prompt, + build_format_selection_prompt, + build_volume_organization_prompt, + get_certainty_assessment_json_schema, + get_certainty_assessment_system_prompt, + get_certainty_validator_json_schema, + get_format_selection_json_schema, + get_format_selection_system_prompt, + get_format_validator_json_schema, + get_volume_organization_json_schema, + get_volume_organization_system_prompt, + get_volume_validator_json_schema, +) +from backend.autonomous.prompts.paper_redundancy_prompts import ( + build_paper_redundancy_prompt, +) +from backend.shared.config import system_config +from backend.shared.models import VolumeChapter, VolumeOrganization + + +PAPERS = [ + { + "paper_id": "paper_001", + "title": "Mechanism", + "abstract": "A proposed mechanism with simulation evidence.", + "outline": "Mechanism; constraints; validation", + "word_count": 1200, + } +] + + +def test_redundancy_preserves_cross_domain_unique_value() -> None: + prompt = build_paper_redundancy_prompt( + "Design a safer low-energy desalination system.", + PAPERS, + ).lower() + + for concept in ( + "solution mechanism", + "evidence", + "implementation", + "experimental proposal", + "failure-mode", + "theorem", + "proof", + "algorithm", + "impossibility result", + ): + assert concept in prompt + assert "at most one" in prompt + assert "when in doubt, do not recommend removal" in prompt + + +def test_certainty_preserves_evidence_status_and_generalized_impossibility() -> None: + prompt = get_certainty_assessment_system_prompt().lower() + + assert "proposals" in prompt + assert "hypotheses" in prompt + assert "required validation" in prompt + assert "never treat an invention" in prompt + for impossibility in ( + "mathematically impossible", + "physically infeasible", + "internally inconsistent", + "prohibited by stated constraints", + ): + assert impossibility in prompt + + +def test_certainty_builder_handles_invention_without_claiming_demonstration() -> None: + prompt = build_certainty_assessment_prompt( + "Invent a compact atmospheric water harvester.", + PAPERS, + ).lower() + + assert "strongest defensible answer" in prompt + assert "proposed mechanism with simulation evidence" in prompt + assert "proposals" in prompt + assert "demonstrated merely because it is proposed" in prompt + + +def test_certainty_expansion_supplements_complete_library_catalog() -> None: + papers = PAPERS + [ + { + "paper_id": "paper_002", + "title": "Limitation", + "abstract": "A contradiction and unvalidated assumption.", + "outline": "Limitations", + "word_count": 600, + } + ] + prompt = build_certainty_assessment_prompt( + "Assess the proposed system.", + papers, + expanded_papers=[ + { + "paper_id": "paper_001", + "title": "Mechanism", + "content": "Full mechanism evidence", + } + ], + ) + + assert "paper_001" in prompt + assert "paper_002" in prompt + assert "A contradiction and unvalidated assumption." in prompt + assert "Full mechanism evidence" in prompt + + +def test_certainty_validator_receives_evidence_not_only_titles() -> None: + prompt = build_certainty_validation_prompt( + "Assess the proposed system.", + PAPERS, + { + "certainty_level": "partial_answer", + "known_certainties_summary": "The mechanism is proposed.", + "reasoning": "No prototype exists.", + }, + expanded_papers=[ + { + "paper_id": "paper_001", + "title": "Mechanism", + "content": "The theorem is proved, but the prototype is unbuilt.", + } + ], + ) + + assert "A proposed mechanism with simulation evidence." in prompt + assert "Mechanism; constraints; validation" in prompt + assert "The theorem is proved, but the prototype is unbuilt." in prompt + + +def test_certainty_expansion_recognizes_all_solution_modalities() -> None: + assessor = CertaintyAssessor("submitter", "validator") + prompt = assessor._build_expansion_request_prompt("Solve the objective.", PAPERS).lower() + + for concept in ( + "mechanism", + "evidence", + "implementation", + "risk", + "validation", + "theorem", + "proof", + ): + assert concept in prompt + assert '"expand_papers"' in prompt + assert '"proceed_without_expansion"' in prompt + + +def test_format_selection_is_domain_neutral_and_not_paper_count_driven() -> None: + prompt = get_format_selection_system_prompt().lower() + + assert "genuinely independent solution components" in prompt + assert "number of source papers alone does not justify a volume" in prompt + assert "mathematical" in prompt + assert "formal proof" in prompt + + +def test_volume_supports_cross_domain_dependency_structures_and_gaps() -> None: + prompt = get_volume_organization_system_prompt().lower() + schema = get_volume_organization_json_schema().lower() + + for concept in ( + "foundations → results → proofs", + "constraints → mechanism → implementation → validation", + "hypothesis → evidence → protocol → limitations", + "design", + "evaluation", + "safety", + "risk", + ): + assert concept in prompt + assert "mixed engineering and formal analysis" in schema + assert "mathematical bounds" in schema + assert "without claiming unperformed tests" in schema + + +def test_existing_tier3_json_contracts_are_unchanged() -> None: + certainty = get_certainty_assessment_json_schema() + assert '"certainty_level"' in certainty + assert '"known_certainties_summary"' in certainty + assert '"reasoning"' in certainty + for value in ( + "total_answer", + "partial_answer", + "no_answer_known", + "appears_impossible", + "other", + ): + assert value in certainty + + fmt = get_format_selection_json_schema() + assert '"answer_format"' in fmt + assert "short_form | long_form" in fmt + assert '"decision"' in get_format_validator_json_schema() + assert '"decision"' in get_certainty_validator_json_schema() + + volume = get_volume_organization_json_schema() + for field in ( + '"volume_title"', + '"chapters"', + '"chapter_type"', + '"paper_id"', + '"title"', + '"order"', + '"description"', + '"outline_complete"', + '"reasoning"', + ): + assert field in volume + for value in ("existing_paper", "gap_paper", "introduction", "conclusion"): + assert value in volume + assert '"decision"' in get_volume_validator_json_schema() + + +def test_active_builders_keep_schema_before_untrusted_context() -> None: + certainty = build_certainty_assessment_prompt("UNTRUSTED_GOAL", PAPERS) + fmt = build_format_selection_prompt( + "UNTRUSTED_GOAL", + PAPERS, + {"certainty_level": "partial_answer", "known_certainties_summary": "supported"}, + ) + volume = build_volume_organization_prompt( + "UNTRUSTED_GOAL", + PAPERS, + {"certainty_level": "partial_answer", "known_certainties_summary": "supported"}, + ) + + for prompt in (certainty, fmt, volume): + assert prompt.index("REQUIRED JSON FORMAT") < prompt.index("UNTRUSTED_GOAL") + + +@pytest.mark.parametrize( + ("title", "findings", "context", "expected"), + [ + ( + "A Testable Atmospheric Water Harvester", + "Simulation supports the mechanism; prototype performance is unvalidated.", + "", + "prototype performance is unvalidated", + ), + ( + "A New Extremal Bound", + "The theorem and Lean-verified proof establish the requested bound.", + "", + "lean-verified proof", + ), + ( + "A Safe Controller with Formal Limits", + "Engineering evidence supports the controller; a theorem bounds instability.", + "Chapter type: gap_paper. Close the implementation and safety-validation gap.", + "close the implementation and safety-validation gap", + ), + ], +) +def test_tier3_compiler_prompt_assembly_is_domain_general_and_proof_aware( + monkeypatch, + title, + findings, + context, + expected, +) -> None: + coordinator = AutonomousCoordinator() + seen = {} + + def apply_proof_context(prompt: str) -> str: + seen["prompt"] = prompt + return f"{prompt}\nPROOF_CONTEXT_APPLIED" + + monkeypatch.setattr(coordinator, "_apply_proof_context", apply_proof_context) + prompt = coordinator._build_tier3_compiler_prompt(title, findings, context) + lowered = prompt.lower() + + assert "write a mathematical research paper" not in lowered + assert "strongest credible and genuinely novel solution" in lowered + assert "domain" in lowered + assert "mathematical reasoning" in lowered + assert "formal proof" in lowered + assert expected in lowered + if context: + assert "directly advances the final answer" in lowered + assert "directly answers the research question" not in lowered + else: + assert "directly answers the research question" in lowered + assert prompt.endswith("PROOF_CONTEXT_APPLIED") + assert seen["prompt"] in prompt + + +@pytest.mark.asyncio +async def test_tier3_reference_selection_passes_six_paper_cap(monkeypatch) -> None: + coordinator = AutonomousCoordinator() + seen = {} + + async def select_references(**kwargs): + seen.update(kwargs) + return ["paper_001"] + + monkeypatch.setattr( + coordinator, + "_get_effective_user_research_prompt", + lambda: "Solve it.", + ) + coordinator._reference_selector = SimpleNamespace( + select_references=select_references + ) + + selected = await coordinator._tier3_reference_selection(PAPERS) + + assert selected == ["paper_001"] + assert seen["max_total_papers"] == 6 + assert seen["available_papers"] == PAPERS + assert seen["already_selected"] == [] + + +@pytest.mark.asyncio +async def test_tier3_title_selection_reuses_exploration_candidates(monkeypatch) -> None: + coordinator = AutonomousCoordinator() + assessment = SimpleNamespace( + certainty_level="partial_answer", + known_certainties_summary="The mechanism is supported; validation remains.", + ) + seen = {} + + monkeypatch.setattr( + coordinator, + "_get_reference_paper_details", + lambda _ids: _async_result(PAPERS), + ) + monkeypatch.setattr( + coordinator, + "_paper_title_exploration_phase", + lambda **_kwargs: _async_result( + "\n".join(f"Candidate {index}" for index in range(1, 6)) + ), + ) + monkeypatch.setattr( + coordinator, + "_get_effective_user_research_prompt", + lambda: "Solve it.", + ) + + async def select_title(**kwargs): + seen.update(kwargs) + return "Validated Final Title" + + coordinator._title_selector = SimpleNamespace(select_title=select_title) + + title = await coordinator._tier3_title_selection(assessment, ["paper_001"]) + + assert title == "Validated Final Title" + assert seen["candidate_titles"].count("Candidate") == 5 + assert seen["reference_papers"] == PAPERS + assert "Defensible Findings and Evidence Status" in seen["brainstorm_summary"] + + +@pytest.mark.asyncio +async def test_volume_chapter_forwards_role_scope_and_all_existing_references( + monkeypatch, +) -> None: + coordinator = AutonomousCoordinator() + chapter = VolumeChapter( + chapter_type="gap_paper", + title="Validation Gap", + order=3, + description="Build and validate the fail-safe controller.", + ) + volume = VolumeOrganization( + volume_title="Mixed solution", + chapters=[ + VolumeChapter( + chapter_type="existing_paper", + paper_id="paper_001", + title="Mechanism", + order=2, + ), + chapter, + ], + ) + assessment = SimpleNamespace( + known_certainties_summary="The mechanism is supported; validation remains." + ) + seen = {} + + monkeypatch.setattr( + coordinator, + "_get_reference_paper_details", + lambda _ids: _async_result(PAPERS), + ) + monkeypatch.setattr( + coordinator, + "_paper_title_exploration_phase", + lambda **_kwargs: _async_result("Candidate One"), + ) + coordinator._title_selector = SimpleNamespace( + select_title=lambda **_kwargs: _async_result("Validated Gap Chapter") + ) + + async def compile_paper(**kwargs): + seen.update(kwargs) + return "Abstract\nComplete chapter" + + monkeypatch.setattr(coordinator, "_compile_tier3_paper", compile_paper) + coordinator_module = importlib.import_module( + "backend.autonomous.core.autonomous_coordinator" + ) + monkeypatch.setattr( + coordinator_module.outline_memory, + "get_outline", + lambda: _async_result("Outline"), + ) + monkeypatch.setattr( + coordinator_module.final_answer_memory, + "save_chapter_paper", + lambda **_kwargs: _async_result(None), + ) + monkeypatch.setattr( + coordinator_module.final_answer_memory, + "get_chapter_paper", + lambda _order: _async_result(""), + ) + + assert await coordinator._write_volume_chapter(chapter, volume, assessment) + assert seen["reference_paper_ids"] == ["paper_001"] + assert "gap_paper" in seen["writing_context"] + assert "Build and validate the fail-safe controller." in seen["writing_context"] + assert system_config.autonomous_tier3_short_form_max_reference_papers == 6 + + +@pytest.mark.asyncio +async def test_volume_chapter_requires_validated_final_title(monkeypatch) -> None: + coordinator = AutonomousCoordinator() + chapter = VolumeChapter( + chapter_type="gap_paper", + title="Organizer Draft", + order=2, + description="Close the validation gap.", + ) + volume = VolumeOrganization(volume_title="Volume", chapters=[chapter]) + assessment = SimpleNamespace(known_certainties_summary="Supported findings") + compiled = False + + monkeypatch.setattr( + coordinator, + "_get_reference_paper_details", + lambda _ids: _async_result([]), + ) + monkeypatch.setattr( + coordinator, + "_paper_title_exploration_phase", + lambda **_kwargs: _async_result("Five candidate titles"), + ) + coordinator._title_selector = SimpleNamespace( + select_title=lambda **_kwargs: _async_result(None) + ) + + async def compile_paper(**_kwargs): + nonlocal compiled + compiled = True + return "Should not run" + + monkeypatch.setattr(coordinator, "_compile_tier3_paper", compile_paper) + + assert not await coordinator._write_volume_chapter(chapter, volume, assessment) + assert not compiled + + +@pytest.mark.asyncio +async def test_generated_gap_content_reaches_later_conclusion(monkeypatch) -> None: + coordinator = AutonomousCoordinator() + gap = VolumeChapter( + chapter_type="gap_paper", + title="Generated Gap", + order=2, + description="Close the implementation gap.", + ) + conclusion = VolumeChapter( + chapter_type="conclusion", + title="Conclusion", + order=3, + ) + volume = VolumeOrganization( + volume_title="Volume", + chapters=[gap, conclusion], + ) + assessment = SimpleNamespace(known_certainties_summary="Supported findings") + seen = {} + + monkeypatch.setattr( + coordinator, + "_get_reference_paper_details", + lambda _ids: _async_result([]), + ) + monkeypatch.setattr( + coordinator, + "_paper_title_exploration_phase", + lambda **_kwargs: _async_result("Five candidate titles"), + ) + coordinator._title_selector = SimpleNamespace( + select_title=lambda **_kwargs: _async_result("Validated Conclusion") + ) + coordinator_module = importlib.import_module( + "backend.autonomous.core.autonomous_coordinator" + ) + monkeypatch.setattr( + coordinator_module.final_answer_memory, + "get_chapter_paper", + lambda order: _async_result( + "Generated implementation and validation evidence" if order == 2 else "" + ), + ) + monkeypatch.setattr( + coordinator_module.outline_memory, + "get_outline", + lambda: _async_result("Outline"), + ) + monkeypatch.setattr( + coordinator_module.final_answer_memory, + "save_chapter_paper", + lambda **_kwargs: _async_result(None), + ) + + async def compile_paper(**kwargs): + seen.update(kwargs) + return "Abstract\nConclusion" + + monkeypatch.setattr(coordinator, "_compile_tier3_paper", compile_paper) + + assert await coordinator._write_volume_chapter(conclusion, volume, assessment) + assert "Generated implementation and validation evidence" in seen[ + "generated_chapter_context" + ] + + +async def _async_result(value): + return value + + +def test_volume_writing_order_remains_gap_conclusion_introduction() -> None: + volume = VolumeOrganization( + volume_title="Mixed solution", + chapters=[ + VolumeChapter( + chapter_type="introduction", + title="Introduction", + order=1, + ), + VolumeChapter( + chapter_type="gap_paper", + title="Validation", + order=3, + ), + VolumeChapter( + chapter_type="existing_paper", + paper_id="paper_001", + title="Mechanism", + order=2, + ), + VolumeChapter( + chapter_type="conclusion", + title="Conclusion", + order=4, + ), + ], + ) + + order = VolumeOrganizer("submitter", "validator").get_writing_order(volume) + assert [chapter.chapter_type for chapter in order] == [ + "gap_paper", + "conclusion", + "introduction", + ] diff --git a/tests/test_assistant_memory_injection.py b/tests/unit/test_assistant_memory_injection.py similarity index 81% rename from tests/test_assistant_memory_injection.py rename to tests/unit/test_assistant_memory_injection.py index 0f96ee6..4da36c6 100644 --- a/tests/test_assistant_memory_injection.py +++ b/tests/unit/test_assistant_memory_injection.py @@ -99,6 +99,36 @@ async def test_creative_producer_gets_latest_compatible_memory_without_blocking( self.assertIn("OPTIONAL ASSISTANT MEMORY CONTEXT", messages[0]["content"]) self.assertIn("ASSISTANT RETRIEVED MEMORY SUPPORT", messages[0]["content"]) + async def test_formalization_role_does_not_refresh_assistant_memory(self) -> None: + pack = AssistantProofPack( + workflow_mode="autonomous", + target_kind="proof_candidate", + target_hash="old_target_hash", + results=[_support()], + ) + fake_assistant = _FakeAssistantCoordinator(pack) + manager_module.assistant_proof_search_coordinator = fake_assistant + manager = APIClientManager() + prompt = ( + "USER PROMPT:\nProve useful facts about True.\n\n" + "THEOREM CANDIDATE:\ntheorem target : True\n\n" + "YOUR TASK:\nReturn Lean 4 proof JSON." + ) + + messages, consumed_hash = await manager._maybe_add_assistant_memory_context( + task_id="proof_form_001", + role_id="autonomous_proof_formalization_paper_1", + role_config=ModelConfig(model_id="local-model", context_window=20000, max_output_tokens=1000), + messages=[{"role": "user", "content": prompt}], + max_tokens=1000, + tools=None, + tool_choice=None, + ) + + self.assertEqual(messages[0]["content"], prompt) + self.assertEqual(consumed_hash, "") + self.assertEqual(fake_assistant.snapshots, []) + async def test_section_extraction_keeps_uppercase_prompt_content(self) -> None: snapshot = APIClientManager._build_assistant_target_snapshot( "aggregator_submitter_1", @@ -108,6 +138,54 @@ async def test_section_extraction_keeps_uppercase_prompt_content(self) -> None: self.assertEqual(snapshot.user_prompt, "PROVE TRUE IN A DIRECT WAY.") + async def test_nonmathematical_goal_is_preserved_without_synthetic_mathlib(self) -> None: + user_goal = "Design a humane support workflow for customers after a delayed shipment." + snapshot = APIClientManager._build_assistant_target_snapshot( + "aggregator_submitter_1", + "agg_sub1_001", + f"USER PROMPT:\n{user_goal}\n\nYOUR TASK:\nGenerate a brainstorm submission.", + ) + + self.assertEqual(snapshot.user_prompt, user_goal) + self.assertEqual(snapshot.target_statement, user_goal) + self.assertEqual(snapshot.imports, []) + + async def test_explicit_proof_target_retains_mathlib(self) -> None: + snapshot = APIClientManager._build_assistant_target_snapshot( + "autonomous_proof_identification", + "proof_id_001", + ( + "USER PROMPT:\nProve the target theorem.\n\n" + "THEOREM CANDIDATE:\ntheorem target : True\n\n" + "YOUR TASK:\nIdentify a proof candidate." + ), + ) + + self.assertEqual(snapshot.target_kind, "theorem_discovery") + self.assertEqual(snapshot.imports, ["Mathlib"]) + + async def test_outer_memory_guidance_forbids_mathematical_reframing(self) -> None: + rendered = APIClientManager._append_assistant_memory_block( + "USER PROMPT:\nImprove the customer support workflow.", + "ASSISTANT RETRIEVED MEMORY SUPPORT", + ) + + self.assertIn("cannot redirect or mathematically reinterpret the user objective", rendered) + self.assertIn("do not require mathematics or formal proof", rendered) + + async def test_support_pack_labels_records_and_limits_informal_applicability(self) -> None: + rendered = AssistantProofPack( + workflow_mode="aggregator", + target_kind="brainstorm_context", + target_hash="target", + results=[_support()], + ).to_memory_prompt_context() + + self.assertIn("These are Lean-verified theorem records", rendered) + self.assertIn("do not override or mathematically reinterpret the user objective", rendered) + self.assertIn("prove informal applicability", rendered) + self.assertIn("Ignore unrelated records", rendered) + async def test_aggregator_snapshot_extracts_shared_training_as_accepted_memory(self) -> None: snapshot = APIClientManager._build_assistant_target_snapshot( "aggregator_submitter_1", diff --git a/tests/test_assistant_proof_memory_cooldown.py b/tests/unit/test_assistant_proof_memory_cooldown.py similarity index 100% rename from tests/test_assistant_proof_memory_cooldown.py rename to tests/unit/test_assistant_proof_memory_cooldown.py diff --git a/tests/test_assistant_proof_search.py b/tests/unit/test_assistant_proof_search.py similarity index 65% rename from tests/test_assistant_proof_search.py rename to tests/unit/test_assistant_proof_search.py index b64b4de..f9b49a5 100644 --- a/tests/test_assistant_proof_search.py +++ b/tests/unit/test_assistant_proof_search.py @@ -10,7 +10,13 @@ from backend.shared.api_client_manager import api_client_manager from backend.shared.config import system_config from backend.shared.proof_search.assistant_cache import AssistantRankCache -from backend.shared.proof_search.assistant_coordinator import AssistantProofSearchCoordinator +from backend.shared.proof_search.assistant_coordinator import ( + AssistantProofSearchCoordinator, + _build_assistant_selection_prompt, + _drop_current_run_supports_from_pack, + _filter_assistant_lane_records, + _fuse_assistant_retrieval_lanes, +) from backend.shared.proof_search.assistant_models import ( AssistantProofPack, AssistantProofSupport, @@ -21,6 +27,7 @@ from backend.shared.proof_search.models import ProofSearchRequest, ProofSearchResponse, UnifiedProofSearchRecord from backend.shared.models import ModelConfig from backend.autonomous.memory.session_manager import session_manager +from backend.shared.proof_search import moto_sources def _record( @@ -31,6 +38,7 @@ def _record( corpus: str = "moto", corpus_scope: str = "history", session_id: str = "", + run_id: str = "", source_kind: str = "verified_proof", verified: bool = True, lean_code: str | None = None, @@ -45,6 +53,7 @@ def _record( proof_id=f"proof_{index}", external_fingerprint=fingerprint or f"fp_{index}", session_id=session_id, + run_id=run_id, source_title=f"Proof Source {index}", display_title=f"Helper {index}", theorem_name=f"Assistant.Helper{index}", @@ -90,23 +99,38 @@ async def search_candidate_pool( pool_limit: int, exclude_corpus_scopes: list[str] | None = None, exclude_session_ids: list[str] | None = None, + exclude_run_ids: list[str] | None = None, ) -> list[UnifiedProofSearchRecord]: self.requests.append(request) self.candidate_pool_kwargs.append( { "exclude_corpus_scopes": list(exclude_corpus_scopes or []), "exclude_session_ids": list(exclude_session_ids or []), + "exclude_run_ids": list(exclude_run_ids or []), } ) excluded_scopes = set(exclude_corpus_scopes or []) excluded_sessions = set(exclude_session_ids or []) + excluded_runs = set(exclude_run_ids or []) return [ record for record in self.records - if record.corpus_scope not in excluded_scopes + if record.corpus in request.corpora + and record.corpus_scope not in excluded_scopes and record.session_id not in excluded_sessions + and record.run_id not in excluded_runs ][:pool_limit] + async def exact_identity_neighborhood(self, **kwargs) -> list[UnifiedProofSearchRecord]: + excluded_runs = set(kwargs.get("exclude_run_ids") or []) + excluded_sessions = set(kwargs.get("exclude_session_ids") or []) + return [ + record for record in self.records + if record.corpus in set(kwargs.get("corpora") or []) + and record.run_id not in excluded_runs + and record.session_id not in excluded_sessions + ] + async def _fake_assistant_selector( snapshot: AssistantTargetSnapshot, @@ -121,6 +145,483 @@ async def _fake_assistant_selector( ) +class AssistantProofPackPayloadTests(unittest.IsolatedAsyncioTestCase): + async def test_selector_prompt_requires_concrete_transfer_and_allows_empty_result(self) -> None: + snapshot = AssistantTargetSnapshot( + workflow_mode="aggregator", + target_kind="brainstorm_context", + user_prompt="Improve a warehouse picking workflow.", + target_statement="Improve a warehouse picking workflow.", + ) + support = AssistantProofSupport.from_record(_record(1, corpus="manual")) + + prompt = _build_assistant_selection_prompt(snapshot, [support]) + + self.assertIn("concrete transfer path", prompt) + self.assertIn("Shared keywords and loose analogies are insufficient", prompt) + self.assertIn("Never reinterpret or narrow the target", prompt) + self.assertIn("For a non-mathematical target, use []", prompt) + self.assertIn("bound, invariant, impossibility result", prompt) + self.assertIn("description: Uses trivial to prove True.", prompt) + self.assertIn("source: Proof Source 1", prompt) + self.assertIn("dependencies: True.intro, Dep.1", prompt) + self.assertIn("imports: Mathlib", prompt) + + async def test_support_lineage_is_bounded_metadata_only(self) -> None: + with _assistant_test_environment(): + coordinator = AssistantProofSearchCoordinator(service=_FakeProofSearchService([])) + record = _record(1, corpus="manual") + occurrences = [ + { + "search_id": f"manual:run-{index}:proof_1", + "corpus": "manual", + "run_id": f"run-{index}", + "source_title": f"Source {index}", + } + for index in range(3) + ] + record = record.model_copy(update={"metadata": { + "assistant_occurrences": occurrences, + "assistant_occurrence_total": 3, + }}) + support = AssistantProofSupport.from_record(record) + target_hash = "target-lineage" + coordinator._packs[target_hash] = AssistantProofPack( + workflow_mode="autonomous", + target_kind="proof_candidate", + target_hash=target_hash, + results=[support], + ) + + payload = coordinator.get_support_lineage( + target_hash=target_hash, + support_search_id=support.search_id, + offset=1, + limit=1, + ) + + self.assertEqual(payload["occurrence_total"], 3) + self.assertEqual(payload["next_offset"], 2) + self.assertEqual(payload["occurrences"][0]["run_id"], "run-1") + self.assertNotIn("lean_code", payload["occurrences"][0]) + self.assertNotIn("user_prompt", payload["occurrences"][0]) + + async def test_manual_history_hydrates_detail_before_normalization(self) -> None: + summary = { + "session_id": "manual-run", + "proof_id": "proof-1", + "theorem_statement": "theorem archived : True", + } + detail = { + "lean_code": "import Mathlib\n theorem archived : True := by trivial", + "dependencies": [{"kind": "mathlib", "name": "True.intro"}], + } + with mock.patch.object( + moto_sources.manual_proof_database, + "list_proof_library_from_history", + mock.AsyncMock(return_value=[summary]), + ), mock.patch.object( + moto_sources.manual_proof_database, + "get_library_proof_from_history", + mock.AsyncMock(return_value=detail), + ): + records = await moto_sources._records_from_manual_history() + + self.assertEqual(len(records), 1) + self.assertTrue(records[0].lean_code) + self.assertEqual(records[0].dependency_names, ["True.intro"]) + self.assertTrue(records[0].lean_code_hash) + + async def test_fusion_counts_one_best_vote_per_artifact_per_lane(self) -> None: + snapshot = AssistantTargetSnapshot( + workflow_mode="autonomous", + target_kind="proof_candidate", + user_prompt="Prove True", + target_statement="theorem target : True", + ) + first = _record(1, search_id="moto:first") + duplicate = _record(1, search_id="manual:duplicate", corpus="manual") + second_lane = _record(1, search_id="leanoj:third", corpus="leanoj") + fused, _ = _fuse_assistant_retrieval_lanes( + {"local_history": [first, duplicate], "exact_cross_prompt": [second_lane]}, + snapshot, + limit=64, + ) + + self.assertEqual(len(fused), 1) + self.assertEqual( + fused[0].metadata["assistant_reciprocal_rank_score"], + 2.0, + ) + self.assertEqual(fused[0].metadata["assistant_occurrence_total"], 3) + + async def test_assistant_filters_only_explicit_duplicate_emphasis_artifacts(self) -> None: + ordinary_duplicate_novel = _record(1).model_copy( + update={"novelty_tier": "duplicate_novel"} + ) + excluded_emphasis = _record(2).model_copy( + update={ + "novelty_tier": "duplicate_novel", + "metadata": { + "assistant_exclude_standalone_exact_duplicate_emphasis": True, + }, + } + ) + + filtered = _filter_assistant_lane_records( + [ordinary_duplicate_novel, excluded_emphasis] + ) + + self.assertEqual([record.search_id for record in filtered], [ordinary_duplicate_novel.search_id]) + + async def test_cached_pack_drops_explicit_duplicate_emphasis_support(self) -> None: + ordinary = AssistantProofSupport.from_record( + _record(1).model_copy(update={"novelty_tier": "duplicate_novel"}) + ) + excluded = AssistantProofSupport.from_record( + _record(2).model_copy( + update={ + "novelty_tier": "duplicate_novel", + "metadata": { + "assistant_exclude_standalone_exact_duplicate_emphasis": True, + }, + } + ) + ) + pack = AssistantProofPack( + workflow_mode="autonomous", + target_kind="proof_candidate", + target_hash="cached-target", + results=[ordinary, excluded], + ) + + sanitized = _drop_current_run_supports_from_pack(pack) + + self.assertEqual([support.search_id for support in sanitized.results], [ordinary.search_id]) + + async def test_cached_pack_keeps_unmarked_occurrence_from_mixed_support(self) -> None: + support = AssistantProofSupport.from_record(_record(1)).model_copy( + update={ + "occurrence_provenance": [ + { + "search_id": "manual:marked", + "corpus": "manual", + "run_id": "marked-run", + "assistant_exclude_standalone_exact_duplicate_emphasis": "true", + }, + { + "search_id": "moto:eligible", + "corpus": "moto", + "run_id": "eligible-run", + }, + ], + "occurrence_total": 2, + } + ) + pack = AssistantProofPack( + workflow_mode="autonomous", + target_kind="proof_candidate", + target_hash="mixed-target", + results=[support], + ) + + sanitized = _drop_current_run_supports_from_pack(pack) + + self.assertEqual(len(sanitized.results), 1) + retained = sanitized.results[0] + self.assertEqual(retained.search_id, "moto:eligible") + self.assertEqual(retained.run_id, "eligible-run") + self.assertEqual(retained.occurrence_total, 1) + self.assertEqual(len(retained.occurrence_provenance), 1) + + async def test_cached_pack_filters_excluded_run_per_occurrence(self) -> None: + support = AssistantProofSupport.from_record(_record(1)).model_copy( + update={ + "run_id": "eligible-run", + "occurrence_provenance": [ + { + "search_id": "moto:eligible", + "corpus": "moto", + "run_id": "eligible-run", + }, + { + "search_id": "manual:current", + "corpus": "manual", + "run_id": "current-run", + }, + ], + "occurrence_total": 2, + } + ) + pack = AssistantProofPack( + workflow_mode="autonomous", + target_kind="proof_candidate", + target_hash="run-filter-target", + results=[support], + ) + + sanitized = _drop_current_run_supports_from_pack( + pack, + excluded_run_ids=["current-run"], + ) + + self.assertEqual(len(sanitized.results), 1) + retained = sanitized.results[0] + self.assertEqual(retained.occurrence_total, 1) + self.assertEqual( + [occurrence["run_id"] for occurrence in retained.occurrence_provenance], + ["eligible-run"], + ) + + async def test_current_run_excluded_for_autonomous_manual_and_leanoj(self) -> None: + with _assistant_test_environment(): + for mode, run_id in ( + ("autonomous", "auto-session-1"), + ("manual_proof_check", "manual-run-1"), + ("leanoj", "leanoj-session-1"), + ): + service = _FakeProofSearchService([ + _record(1, corpus="manual", run_id=run_id), + _record(2, corpus="manual", run_id=f"{run_id}-history"), + ]) + coordinator = AssistantProofSearchCoordinator( + service=service, + assistant_selector=_fake_assistant_selector, + ) + pack = await coordinator.refresh_now(AssistantTargetSnapshot( + workflow_mode=mode, + target_kind="proof_candidate" if mode != "leanoj" else "final_solver", + user_prompt="Prove True.", + target_statement="theorem target : True", + run_id=run_id, + )) + self.assertIsNotNone(pack) + self.assertEqual([result.run_id for result in pack.results], [f"{run_id}-history"]) + self.assertTrue(all(run_id in kwargs["exclude_run_ids"] for kwargs in service.candidate_pool_kwargs)) + await coordinator.stop_all(clear_packs=True) + + async def test_cached_pack_filters_active_run(self) -> None: + with _assistant_test_environment(): + coordinator = AssistantProofSearchCoordinator(service=_FakeProofSearchService([])) + snapshot = AssistantTargetSnapshot( + workflow_mode="manual_proof_check", + target_kind="proof_candidate", + user_prompt="Prove True.", + run_id="manual-active", + ) + target_hash = snapshot.stable_hash() + snapshot = snapshot.model_copy(update={"target_hash": target_hash}) + persisted = AssistantProofPack( + workflow_mode=snapshot.workflow_mode, + target_kind=snapshot.target_kind, + target_hash=target_hash, + freshness="cached", + selection_mode="cached", + results=[ + AssistantProofSupport.from_record(_record(1, corpus="manual", run_id="manual-active")), + AssistantProofSupport.from_record(_record(2, corpus="manual", run_id="manual-history")), + ], + ) + cached = _drop_current_run_supports_from_pack( + AssistantProofPack.model_validate_json(persisted.model_dump_json()), + excluded_run_ids=[snapshot.run_id], + ) + self.assertEqual([result.run_id for result in cached.results], ["manual-history"]) + + async def test_filtered_empty_cached_pack_schedules_retrieval_for_stable_workflow_ids(self) -> None: + with _assistant_test_environment(): + for mode, stable_id, identity_field in ( + ("autonomous", "auto-session-active", "session_id"), + ("manual_proof_check", "manual-run-active", "run_id"), + ("leanoj", "leanoj-session-active", "session_id"), + ): + snapshot = AssistantTargetSnapshot( + workflow_mode=mode, + target_kind="final_solver" if mode == "leanoj" else "proof_candidate", + user_prompt=f"Find supports for {mode}.", + target_statement="theorem target : True", + run_id=stable_id, + ) + excluded_record = _record( + 1, + corpus="manual", + **{identity_field: stable_id}, + ) + history_record = _record( + 2, + corpus="manual", + run_id=f"{stable_id}-history", + ) + coordinator = AssistantProofSearchCoordinator( + service=_FakeProofSearchService([history_record]), + assistant_selector=_fake_assistant_selector, + ) + target_hash = snapshot.stable_hash() + cached_pack = AssistantProofPack( + workflow_mode=mode, + target_kind=snapshot.target_kind, + target_hash=target_hash, + freshness="cached", + selection_mode="cached", + results=[AssistantProofSupport.from_record(excluded_record)], + ) + coordinator._cache.record_pack( + snapshot=snapshot.model_copy(update={"target_hash": target_hash}), + pack=cached_pack, + selected_search_ids=[excluded_record.search_id], + ) + + submitted_hash = coordinator.submit_target(snapshot) + + self.assertEqual(submitted_hash, target_hash) + self.assertIn(target_hash, coordinator._tasks) + await coordinator._tasks[target_hash] + refreshed = coordinator.get_latest_pack(target_hash) + self.assertEqual( + [support.run_id for support in refreshed.results], + [f"{stable_id}-history"], + ) + await coordinator.stop_all(clear_packs=True) + + async def test_stale_two_read_reuse_filters_requesting_snapshot_before_deferral(self) -> None: + with _assistant_test_environment(): + for mode, stable_id, identity_field in ( + ("autonomous", "auto-session-active", "session_id"), + ("manual_proof_check", "manual-run-active", "run_id"), + ("leanoj", "leanoj-session-active", "session_id"), + ): + coordinator = AssistantProofSearchCoordinator( + service=_FakeProofSearchService([ + _record(3, corpus="manual", run_id=f"{stable_id}-new-history"), + ]), + assistant_selector=_fake_assistant_selector, + ) + old_hash = f"{mode}-old-target" + coordinator._packs[old_hash] = AssistantProofPack( + workflow_mode=mode, + target_kind="proof_candidate", + target_hash=old_hash, + results=[ + AssistantProofSupport.from_record(_record( + 1, + corpus="manual", + **{identity_field: stable_id}, + )), + AssistantProofSupport.from_record(_record( + 2, + corpus="manual", + run_id=f"{stable_id}-history", + )), + ], + ) + coordinator._latest_pack_target_hash = old_hash + coordinator._latest_pack_consumption_count = 1 + snapshot = AssistantTargetSnapshot( + workflow_mode=mode, + target_kind="final_solver" if mode == "leanoj" else "proof_candidate", + user_prompt=f"Find updated supports for {mode}.", + target_statement="theorem changed_target : True", + run_id=stable_id, + ) + + target_hash = coordinator.submit_target(snapshot) + deferred = coordinator.get_latest_pack(target_hash) + + self.assertEqual(deferred.selection_mode, "stale-but-best-known") + self.assertEqual( + [support.run_id for support in deferred.results], + [f"{stable_id}-history"], + ) + self.assertNotIn(target_hash, coordinator._tasks) + + coordinator.mark_pack_consumed_by_solver(target_hash) + coordinator.submit_target(snapshot) + self.assertIn(target_hash, coordinator._tasks) + await coordinator._tasks[target_hash] + await coordinator.stop_all(clear_packs=True) + + async def test_latest_pack_payload_includes_metadata_without_lean_code(self) -> None: + with _assistant_test_environment(): + service = _FakeProofSearchService([_record(index, corpus="manual") for index in range(1, 4)]) + coordinator = AssistantProofSearchCoordinator( + service=service, + assistant_selector=_fake_assistant_selector, + ) + snapshot = AssistantTargetSnapshot( + workflow_mode="autonomous", + target_kind="proof_candidate", + workflow_phase="brainstorm_proof_verification", + user_prompt="Find proof supports.", + target_statement="theorem target : True", + source_type="brainstorm", + source_id="topic_001", + ) + + pack = await coordinator.refresh_now(snapshot) + payload = coordinator.get_latest_pack_payload() + + self.assertTrue(payload["has_pack"]) + self.assertEqual(payload["result_count"], len(pack.results)) + self.assertEqual(payload["selection_reasoning"], pack.selection_reasoning) + self.assertEqual(len(payload["results"]), 3) + first = payload["results"][0] + self.assertEqual(first["novelty_tier"], "novel_formulation") + self.assertEqual(first["novelty_reasoning"], "Fixture proof.") + self.assertEqual(first["source_title"], "Proof Source 1") + self.assertEqual(first["created_at"], "2026-06-12T00:00:01+00:00") + self.assertEqual(first["lean_code"], "") + self.assertTrue(first["has_hydrated_code"]) + await coordinator.stop_all(clear_packs=True) + + async def test_latest_pack_payload_reads_persisted_metadata_file(self) -> None: + with _assistant_test_environment(): + writer = AssistantProofSearchCoordinator( + service=_FakeProofSearchService([_record(1, corpus="manual")]), + assistant_selector=_fake_assistant_selector, + ) + snapshot = AssistantTargetSnapshot( + workflow_mode="autonomous", + target_kind="proof_candidate", + user_prompt="Find persisted supports.", + target_statement="theorem target : True", + ) + + await writer.refresh_now(snapshot) + reader = AssistantProofSearchCoordinator(service=_FakeProofSearchService([])) + payload = reader.get_latest_pack_payload() + + self.assertTrue(payload["has_pack"]) + self.assertEqual(payload["result_count"], 1) + self.assertEqual(payload["results"][0]["proof_id"], "proof_1") + self.assertEqual(payload["results"][0]["lean_code"], "") + await writer.stop_all(clear_packs=True) + + async def test_latest_pack_payload_does_not_return_persisted_pack_when_disabled(self) -> None: + with _assistant_test_environment(): + writer = AssistantProofSearchCoordinator( + service=_FakeProofSearchService([_record(1, corpus="manual")]), + assistant_selector=_fake_assistant_selector, + ) + snapshot = AssistantTargetSnapshot( + workflow_mode="autonomous", + target_kind="proof_candidate", + user_prompt="Find persisted supports.", + target_statement="theorem target : True", + ) + await writer.refresh_now(snapshot) + + system_config.agent_conversation_memory_enabled = False + reader = AssistantProofSearchCoordinator(service=_FakeProofSearchService([])) + payload = reader.get_latest_pack_payload() + + self.assertFalse(payload["enabled"]) + self.assertFalse(payload["has_pack"]) + self.assertEqual(payload["results"], []) + self.assertEqual(payload["disabled_reason"], "Session History Memory is disabled.") + system_config.agent_conversation_memory_enabled = True + await writer.stop_all(clear_packs=True) + + def _response_json(payload: str) -> dict: return {"choices": [{"message": {"content": payload}}]} @@ -180,6 +681,42 @@ def _assistant_test_environment( class AssistantProofRankerTests(unittest.TestCase): + def test_lane_fusion_caps_without_quotas_and_preserves_occurrences(self) -> None: + target = AssistantTargetSnapshot( + workflow_mode="autonomous", + target_kind="proof_candidate", + user_prompt="Prove helper theorems.", + target_statement="theorem target : True", + ) + dominant = [_record(index, corpus="manual") for index in range(80)] + duplicate = _record( + 999, + search_id="syntheticlib4:duplicate", + corpus="syntheticlib4", + fingerprint="other-fingerprint", + ).model_copy(update={ + "theorem_statement_hash": dominant[0].theorem_statement_hash, + "lean_code_hash": dominant[0].lean_code_hash, + }) + + fused, counts = _fuse_assistant_retrieval_lanes( + {"local_history": dominant, "syntheticlib4": [duplicate]}, + target, + limit=64, + ) + + self.assertEqual(len(fused), 64) + self.assertGreater(sum(record.corpus == "manual" for record in fused), 60) + first = next(record for record in fused if record.theorem_statement_hash == "stmt_hash_0") + self.assertEqual(len(first.metadata["assistant_occurrences"]), 2) + self.assertEqual( + first.metadata["assistant_retrieval_lanes"], + ["local_history", "syntheticlib4"], + ) + self.assertEqual(counts["raw_total"]["total"], 81) + self.assertEqual(counts["deduped_distinct"]["total"], 80) + self.assertEqual(counts["fused_cap_64"]["total"], 64) + def test_ranker_caps_filters_and_dedupes_verified_supports(self) -> None: target = AssistantTargetSnapshot( workflow_mode="autonomous", @@ -286,7 +823,7 @@ def test_internal_candidate_pool_keeps_syntheticlib4_current_scope(self) -> None class AssistantProofCoordinatorTests(unittest.IsolatedAsyncioTestCase): - async def test_assistant_selector_retries_missing_selected_search_ids(self) -> None: + async def test_assistant_selector_makes_exactly_one_call_on_invalid_output(self) -> None: restore = _preserve_role_config("manual_proof_assistant") try: api_client_manager.configure_role( @@ -309,40 +846,25 @@ async def test_assistant_selector_retries_missing_selected_search_ids(self) -> N AssistantProofSupport.from_record(_record(1, corpus="manual")), AssistantProofSupport.from_record(_record(2, corpus="manual")), ] - responses = [ - _response_json('{"reasoning":"missing selected IDs"}'), - _response_json( - '{"selected_search_ids":["manual:helper_2"],"reasoning":"use helper 2"}' - ), - ] - with mock.patch.object( api_client_manager, "generate_completion", - new=mock.AsyncMock(side_effect=responses), + new=mock.AsyncMock(return_value=_response_json('{"reasoning":"missing selected IDs"}')), ) as generate_completion: - selected_ids, reasoning = await coordinator._select_with_assistant( - snapshot, - shortlist, - assistant_role_id="manual_proof_assistant", - assistant_model_id="assistant-model", - task_id="assistant_pack_manual_001", - ) + with self.assertRaises(ValueError): + await coordinator._select_with_assistant( + snapshot, shortlist, + assistant_role_id="manual_proof_assistant", + assistant_model_id="assistant-model", + task_id="assistant_pack_manual_001", + ) - self.assertEqual(selected_ids, ["manual:helper_2"]) - self.assertEqual(reasoning, "use helper 2") - self.assertEqual(generate_completion.await_count, 2) + self.assertEqual(generate_completion.await_count, 1) first_call = generate_completion.await_args_list[0].kwargs - retry_call = generate_completion.await_args_list[1].kwargs self.assertEqual(first_call["max_tokens"], 2048) self.assertEqual(first_call["response_format"], {"type": "json_object"}) self.assertTrue(first_call["_moto_disable_supercharge"]) self.assertEqual(first_call["_moto_reasoning_effort_override"], "none") - self.assertEqual(retry_call["task_id"], "assistant_pack_manual_001_retry") - self.assertEqual(retry_call["response_format"], {"type": "json_object"}) - self.assertTrue(retry_call["_moto_disable_supercharge"]) - self.assertEqual(retry_call["_moto_reasoning_effort_override"], "none") - self.assertIn("invalid", retry_call["messages"][0]["content"]) finally: restore() @@ -390,7 +912,7 @@ async def test_lm_studio_assistant_selector_uses_text_response_format(self) -> N finally: restore() - async def test_assistant_selector_retries_ids_outside_shortlist(self) -> None: + async def test_assistant_selector_rejects_ids_outside_shortlist_without_retry(self) -> None: restore = _preserve_role_config("manual_proof_assistant") try: api_client_manager.configure_role( @@ -413,34 +935,21 @@ async def test_assistant_selector_retries_ids_outside_shortlist(self) -> None: AssistantProofSupport.from_record(_record(1, corpus="manual")), AssistantProofSupport.from_record(_record(2, corpus="manual")), ] - responses = [ - _response_json( - '{"selected_search_ids":["manual:not_in_shortlist"],"reasoning":"use a nearby theorem"}' - ), - _response_json( - '{"selected_search_ids":["manual:helper_1"],"reasoning":"use helper 1"}' - ), - ] - with mock.patch.object( api_client_manager, "generate_completion", - new=mock.AsyncMock(side_effect=responses), + new=mock.AsyncMock(return_value=_response_json( + '{"selected_search_ids":["manual:not_in_shortlist"],"reasoning":"bad"}' + )), ) as generate_completion: - selected_ids, reasoning = await coordinator._select_with_assistant( - snapshot, - shortlist, - assistant_role_id="manual_proof_assistant", - assistant_model_id="assistant-model", - task_id="assistant_pack_manual_001", - ) - - self.assertEqual(selected_ids, ["manual:helper_1"]) - self.assertEqual(reasoning, "use helper 1") - self.assertEqual(generate_completion.await_count, 2) - retry_call = generate_completion.await_args_list[1].kwargs - self.assertEqual(retry_call["task_id"], "assistant_pack_manual_001_retry") - self.assertIn("outside the candidate shortlist", retry_call["messages"][0]["content"]) + with self.assertRaises(ValueError): + await coordinator._select_with_assistant( + snapshot, shortlist, + assistant_role_id="manual_proof_assistant", + assistant_model_id="assistant-model", + task_id="assistant_pack_manual_001", + ) + self.assertEqual(generate_completion.await_count, 1) finally: restore() @@ -502,6 +1011,69 @@ async def test_assistant_selector_canonicalizes_unique_bare_proof_ids(self) -> N finally: restore() + async def test_assistant_selector_canonicalizes_fused_occurrence_search_id(self) -> None: + restore = _preserve_role_config("manual_proof_assistant") + try: + api_client_manager.configure_role( + "manual_proof_assistant", + ModelConfig( + provider="openrouter", + model_id="assistant-model", + context_window=4096, + max_output_tokens=2048, + ), + ) + coordinator = AssistantProofSearchCoordinator(service=_FakeProofSearchService([])) + snapshot = AssistantTargetSnapshot( + workflow_mode="manual_proof_check", + target_kind="proof_candidate", + user_prompt="Prove the manual target.", + target_statement="theorem target : True", + ) + support = AssistantProofSupport.from_record( + _record( + 26, + search_id="manual:representative:proof_026", + corpus="manual", + ) + ).model_copy(update={ + "occurrence_provenance": [ + { + "search_id": "manual:representative:proof_026", + "corpus": "manual", + }, + { + "search_id": "moto:historical_run:proof_026", + "corpus": "moto", + }, + ], + "occurrence_total": 2, + }) + + with mock.patch.object( + api_client_manager, + "generate_completion", + new=mock.AsyncMock( + return_value=_response_json( + '{"selected_search_ids":["moto:historical_run:proof_026"],' + '"reasoning":"use the fused historical occurrence"}' + ) + ), + ) as generate_completion: + selected_ids, reasoning = await coordinator._select_with_assistant( + snapshot, + [support], + assistant_role_id="manual_proof_assistant", + assistant_model_id="assistant-model", + task_id="assistant_pack_manual_001", + ) + + self.assertEqual(selected_ids, ["manual:representative:proof_026"]) + self.assertEqual(reasoning, "use the fused historical occurrence") + self.assertEqual(generate_completion.await_count, 1) + finally: + restore() + async def test_assistant_selector_valid_empty_selection_is_not_retried(self) -> None: restore = _preserve_role_config("manual_proof_assistant") try: @@ -585,7 +1157,7 @@ async def test_assistant_selector_failure_publishes_unavailable_pack(self) -> No self.assertEqual(pack.results, []) self.assertEqual(pack.assistant_role_id, "manual_proof_assistant") self.assertEqual(pack.assistant_model_id, "assistant-model") - self.assertEqual(generate_completion.await_count, 2) + self.assertEqual(generate_completion.await_count, 1) self.assertTrue( any("Assistant LLM selection failed" in warning for warning in pack.warnings) ) @@ -669,6 +1241,47 @@ async def test_refresh_persists_metadata_only_status_and_stop_clears_pack(self) self.assertFalse(persisted_path.exists()) self.assertEqual(coordinator.get_status()["cached_pack_count"], 0) + async def test_generic_snapshot_does_not_add_mathlib_to_search_requests(self) -> None: + with _assistant_test_environment(): + service = _FakeProofSearchService([_record(1, corpus="manual")]) + coordinator = AssistantProofSearchCoordinator( + service=service, + assistant_selector=_fake_assistant_selector, + ) + snapshot = AssistantTargetSnapshot( + workflow_mode="aggregator", + target_kind="brainstorm_context", + user_prompt="Improve a warehouse picking workflow.", + target_statement="Improve a warehouse picking workflow.", + ) + + await coordinator.refresh_now(snapshot) + + self.assertTrue(service.requests) + self.assertTrue(all(request.imports == [] for request in service.requests)) + await coordinator.stop_all(clear_packs=True) + + async def test_explicit_formal_snapshot_keeps_mathlib_in_search_requests(self) -> None: + with _assistant_test_environment(): + service = _FakeProofSearchService([_record(1, corpus="manual")]) + coordinator = AssistantProofSearchCoordinator( + service=service, + assistant_selector=_fake_assistant_selector, + ) + snapshot = AssistantTargetSnapshot( + workflow_mode="manual_proof_check", + target_kind="proof_candidate", + user_prompt="Prove the manual target.", + target_statement="theorem target : True", + imports=["Mathlib"], + ) + + await coordinator.refresh_now(snapshot) + + self.assertTrue(service.requests) + self.assertTrue(all(request.imports == ["Mathlib"] for request in service.requests)) + await coordinator.stop_all(clear_packs=True) + async def test_refresh_excludes_active_run_candidates_before_assistant_selection(self) -> None: with _assistant_test_environment(syntheticlib4_enabled=True, session_id="active_session"): service = _FakeProofSearchService( diff --git a/tests/unit/test_autonomous_acceptance_accounting.py b/tests/unit/test_autonomous_acceptance_accounting.py new file mode 100644 index 0000000..264b2a2 --- /dev/null +++ b/tests/unit/test_autonomous_acceptance_accounting.py @@ -0,0 +1,96 @@ +from types import SimpleNamespace +import importlib + +import pytest + +from backend.autonomous.core.autonomous_coordinator import AutonomousCoordinator +from backend.shared.models import BrainstormMetadata, SubmitterConfig + + +def test_legacy_brainstorm_metadata_uses_retained_count_as_cumulative_fallback(): + metadata = BrainstormMetadata( + topic_id="topic_001", + topic_prompt="test", + submission_count=7, + ) + + assert metadata.submission_count == 7 + assert metadata.total_acceptances == 7 + + +def test_brainstorm_metadata_keeps_retained_and_cumulative_counts_separate(): + metadata = BrainstormMetadata( + topic_id="topic_001", + topic_prompt="test", + submission_count=5, + total_acceptances=9, + ) + + assert metadata.submission_count == 5 + assert metadata.total_acceptances == 9 + + +@pytest.mark.asyncio +async def test_solution_path_reviewer_is_configured_before_registry_acquire(monkeypatch): + import backend.shared.solution_path as solution_path + coordinator_module = importlib.import_module( + "backend.autonomous.core.autonomous_coordinator" + ) + + coordinator = AutonomousCoordinator() + primary = SubmitterConfig( + submitter_id=1, + provider="xai_grok_oauth", + model_id="grok-test", + openrouter_provider="host-test", + openrouter_reasoning_effort="high", + lm_studio_fallback_id="fallback-test", + context_window=32000, + max_output_tokens=4000, + supercharge_enabled=True, + ) + coordinator._submitter_configs = [primary] + coordinator._base_user_research_prompt = "test prompt" + coordinator._solution_path_acceptance_count = 2 + + configured = {} + + def configure_role(role_id, config): + configured[role_id] = config + + monkeypatch.setattr( + coordinator_module.api_client_manager, + "configure_role", + configure_role, + ) + monkeypatch.setattr( + coordinator_module.brainstorm_memory, + "get_all_brainstorms", + lambda: _async_value([]), + ) + + manager = SimpleNamespace( + state=SimpleNamespace(acceptance_count=3), + set_acceptance_count=lambda count: _async_value(None), + ) + + async def acquire(*args, **kwargs): + assert "autonomous_solution_path_reviewer" in configured + return manager + + monkeypatch.setattr(solution_path.solution_path_registry, "acquire", acquire) + await coordinator._initialize_solution_path_manager() + + config = configured["autonomous_solution_path_reviewer"] + assert config.provider == primary.provider + assert config.model_id == primary.model_id + assert config.openrouter_provider == primary.openrouter_provider + assert config.openrouter_reasoning_effort == primary.openrouter_reasoning_effort + assert config.lm_studio_fallback_id == primary.lm_studio_fallback_id + assert config.context_window == primary.context_window + assert config.max_output_tokens == primary.max_output_tokens + assert config.supercharge_enabled is True + + +async def _async_value(value): + return value diff --git a/tests/test_autonomous_proof_rounds.py b/tests/unit/test_autonomous_proof_rounds.py similarity index 60% rename from tests/test_autonomous_proof_rounds.py rename to tests/unit/test_autonomous_proof_rounds.py index 48605b6..634f549 100644 --- a/tests/test_autonomous_proof_rounds.py +++ b/tests/unit/test_autonomous_proof_rounds.py @@ -1,6 +1,7 @@ import unittest from importlib import import_module from types import SimpleNamespace +from unittest.mock import AsyncMock from backend.autonomous.core.autonomous_coordinator import ( _BRAINSTORM_MIN_ACCEPTANCES_BEFORE_HANDOFF, @@ -8,9 +9,15 @@ AutonomousCoordinator, ) from backend.autonomous.core.proof_verification_stage import ProofVerificationStage -from backend.autonomous.prompts.proof_prompts import build_proof_identification_prompt +from backend.autonomous.prompts.proof_prompts import ( + PROOF_FRAMING_CONTEXT, + build_proof_identification_prompt, +) +from backend.autonomous.prompts.topic_exploration_prompts import ( + build_exploration_user_prompt, +) from backend.shared.config import system_config -from backend.shared.models import ProofCandidate, ProofStageResult +from backend.shared.models import ProofAttemptFeedback, ProofCandidate, ProofStageResult coordinator_module = import_module("backend.autonomous.core.autonomous_coordinator") @@ -21,6 +28,9 @@ def __init__(self): self.mark_calls = [] self.clear_calls = [] self.workflow_states = [] + self.proof_framing_states = [] + self.interrupted_workflow = False + self.base_user_prompt = "" async def save_proof_checkpoint(self, checkpoint): self.checkpoint = dict(checkpoint) @@ -65,6 +75,18 @@ async def clear_proof_checkpoint(self, source_type=None, source_id=None): async def save_workflow_state(self, state): self.workflow_states.append(dict(state)) + async def get_workflow_state(self): + return dict(self.workflow_states[-1]) if self.workflow_states else {} + + def has_interrupted_workflow(self): + return self.interrupted_workflow + + async def get_base_user_prompt(self): + return self.base_user_prompt + + async def set_proof_framing_state(self, **state): + self.proof_framing_states.append(dict(state)) + class _FakeProofDatabase: def __init__(self): @@ -80,6 +102,19 @@ async def get_pending_retries(self, *args, **kwargs): return [] +class _FakeAPIClientManager: + def __init__(self, response=None, error=None): + self.response = response + self.error = error + self.calls = [] + + async def generate_completion(self, **kwargs): + self.calls.append(kwargs) + if self.error: + raise self.error + return self.response + + class _FakeBrainstormMemory: async def get_metadata(self, _topic_id): return SimpleNamespace(topic_prompt="Topic prompt", submission_count=0) @@ -116,6 +151,60 @@ async def run(self, **kwargs): error_message="retryable proof stage error", ) + +class _DeferredThenCompleteProofStage: + def __init__(self): + self.calls = [] + + async def run(self, **kwargs): + self.calls.append(kwargs) + candidates = kwargs.get("theorem_candidates") or [ + ProofCandidate(theorem_id="done", statement="Completed theorem."), + ProofCandidate(theorem_id="deferred", statement="Deferred theorem."), + ] + if len(self.calls) == 1: + overflow = ProofAttemptFeedback( + attempt=1, + theorem_id="deferred", + error_output="MANDATORY FULL SOURCE CONTEXT OVERFLOW", + overflow_origin="local_preflight", + prompt_tokens=7000, + max_input_tokens=6000, + ) + await kwargs["checkpoint_callback"]( + { + "source_type": kwargs["source_type"], + "source_id": kwargs["source_id"], + "source_title": kwargs["source_title"], + "trigger": kwargs["trigger"], + "status": "deferred", + "candidates": [ + {"index": 1, "candidate": candidates[0].model_dump(mode="json")}, + {"index": 2, "candidate": candidates[1].model_dump(mode="json")}, + ], + "processed_candidate_ids": ["done"], + "deferred_candidate_ids": ["deferred"], + "attempts_by_candidate": { + "deferred": [overflow.model_dump(mode="json")], + }, + "theorem_names_by_candidate": {}, + "results": [], + "total_candidates": 2, + } + ) + return ProofStageResult( + source_type=kwargs["source_type"], + source_id=kwargs["source_id"], + total_candidates=2, + deferred_candidate_ids=["deferred"], + ) + return ProofStageResult( + source_type=kwargs["source_type"], + source_id=kwargs["source_id"], + total_candidates=1, + verified_count=1, + ) + def _configured_coordinator(fake_stage, *, allow_research_papers=True): coordinator = AutonomousCoordinator() coordinator._allow_mathematical_proofs = True @@ -164,6 +253,274 @@ def test_extract_abstract_from_markdown_heading(self): self.assertEqual(abstract, "Markdown abstract content.") +class AutonomousProofFramingBoundaryTests(unittest.IsolatedAsyncioTestCase): + def setUp(self): + self.old_research_metadata = coordinator_module.research_metadata + self.old_proof_database = coordinator_module.proof_database + self.old_api_client_manager = coordinator_module.api_client_manager + coordinator_module.research_metadata = _FakeResearchMetadata() + coordinator_module.proof_database = _FakeProofDatabase() + + def tearDown(self): + coordinator_module.research_metadata = self.old_research_metadata + coordinator_module.proof_database = self.old_proof_database + coordinator_module.api_client_manager = self.old_api_client_manager + + @staticmethod + def _gate_coordinator(): + coordinator = AutonomousCoordinator() + coordinator._allow_mathematical_proofs = True + coordinator._base_user_research_prompt = "Prove a new combinatorial bound." + coordinator._user_research_prompt = "effective prompt should be reset" + coordinator._submitter_configs = [ + SimpleNamespace( + submitter_id=1, + model_id="first-submitter-model", + context_window=8192, + max_output_tokens=777, + ), + SimpleNamespace( + submitter_id=2, + model_id="second-submitter-model", + context_window=16384, + max_output_tokens=999, + ), + ] + return coordinator + + async def test_positive_gate_persists_context_broadcasts_and_uses_first_submitter(self): + fake_api = _FakeAPIClientManager( + response={ + "choices": [{ + "message": { + "content": ( + '{"is_proof_amenable": true, ' + '"reasoning": "Formal bounds directly serve the goal."}' + ) + } + }] + } + ) + coordinator_module.api_client_manager = fake_api + coordinator = self._gate_coordinator() + events = [] + + async def capture_event(event, data): + events.append((event, data)) + + coordinator.set_broadcast_callback(capture_event) + await coordinator._run_proof_framing_gate() + + self.assertTrue(coordinator._proof_framing_active) + self.assertEqual(coordinator._proof_framing_context, PROOF_FRAMING_CONTEXT) + persisted = coordinator_module.research_metadata.proof_framing_states[-1] + self.assertTrue(persisted["active"]) + self.assertEqual(persisted["context"], PROOF_FRAMING_CONTEXT) + self.assertEqual( + events[-1], + ( + "proof_framing_decided", + { + "is_proof_amenable": True, + "reasoning": "Formal bounds directly serve the goal.", + }, + ), + ) + self.assertEqual(len(fake_api.calls), 1) + call = fake_api.calls[0] + self.assertEqual(call["task_id"], "proof_framing_gate_000") + self.assertEqual(call["role_id"], "autonomous_proof_framing_gate") + self.assertEqual(call["model"], "first-submitter-model") + self.assertEqual(call["max_tokens"], 777) + self.assertEqual(call["temperature"], 0.0) + self.assertIn("Prove a new combinatorial bound.", call["messages"][0]["content"]) + + async def test_negative_gate_persists_inactive_state(self): + coordinator_module.api_client_manager = _FakeAPIClientManager( + response={ + "choices": [{ + "message": { + "content": ( + '{"is_proof_amenable": false, ' + '"reasoning": "The task is purely empirical."}' + ) + } + }] + } + ) + coordinator = self._gate_coordinator() + + await coordinator._run_proof_framing_gate() + + self.assertFalse(coordinator._proof_framing_active) + self.assertEqual(coordinator._proof_framing_context, "") + persisted = coordinator_module.research_metadata.proof_framing_states[-1] + self.assertFalse(persisted["active"]) + self.assertEqual(persisted["context"], "") + self.assertEqual(persisted["reasoning"], "The task is purely empirical.") + + ordinary_prompt = build_exploration_user_prompt( + "Develop a safer desalination control strategy.", + [], + [], + ) + captured_prompt = coordinator._append_proof_framing(ordinary_prompt) + self.assertIn("Develop a safer desalination control strategy.", captured_prompt) + self.assertNotIn(PROOF_FRAMING_CONTEXT, captured_prompt) + self.assertIn( + "mathematics, theorem discovery, proof, and formalization remain first-class " + "whenever relevant", + captured_prompt, + ) + + async def test_fresh_start_awaits_proof_framing_gate_once_before_research(self): + coordinator = self._gate_coordinator() + coordinator._get_resume_point = AsyncMock(return_value=None) + coordinator.refresh_workflow_predictions = AsyncMock() + coordinator._broadcast_stopped_once = AsyncMock() + coordinator._topic_exploration_phase = AsyncMock( + side_effect=AssertionError("fresh-start audit must stop before normal research") + ) + + async def stop_after_gate(): + coordinator._stop_event.set() + + coordinator._run_proof_framing_gate = AsyncMock(side_effect=stop_after_gate) + + await coordinator.start() + + coordinator._get_resume_point.assert_awaited_once_with() + coordinator._run_proof_framing_gate.assert_awaited_once_with() + coordinator._topic_exploration_phase.assert_not_awaited() + + async def test_gate_model_and_parse_failures_use_inactive_fallback_with_reason(self): + for fake_api, expected_reason in ( + (_FakeAPIClientManager(error=RuntimeError("provider unavailable")), "provider unavailable"), + ( + _FakeAPIClientManager( + response={"choices": [{"message": {"content": "not valid json"}}]} + ), + "Proof framing gate failed:", + ), + ): + with self.subTest(expected_reason=expected_reason): + coordinator_module.research_metadata = _FakeResearchMetadata() + coordinator_module.api_client_manager = fake_api + coordinator = self._gate_coordinator() + + await coordinator._run_proof_framing_gate() + + self.assertFalse(coordinator._proof_framing_active) + self.assertEqual(coordinator._proof_framing_context, "") + self.assertIn(expected_reason, coordinator._proof_framing_reasoning) + persisted = coordinator_module.research_metadata.proof_framing_states[-1] + self.assertFalse(persisted["active"]) + self.assertEqual(persisted["reasoning"], coordinator._proof_framing_reasoning) + + async def test_disabled_proof_outputs_skip_gate_model_call(self): + fake_api = _FakeAPIClientManager() + coordinator_module.api_client_manager = fake_api + coordinator = self._gate_coordinator() + coordinator._allow_mathematical_proofs = False + + await coordinator._run_proof_framing_gate() + + self.assertEqual(fake_api.calls, []) + self.assertFalse(coordinator._proof_framing_active) + self.assertEqual(coordinator._proof_framing_context, "") + self.assertEqual( + coordinator._proof_framing_reasoning, + "Mathematical proof outputs are disabled for this run.", + ) + + def test_append_proof_framing_is_idempotent(self): + coordinator = self._gate_coordinator() + coordinator._proof_framing_active = True + coordinator._proof_framing_context = PROOF_FRAMING_CONTEXT + + once = coordinator._append_proof_framing("Research goal.") + twice = coordinator._append_proof_framing(once) + + self.assertEqual(twice, once) + self.assertEqual(twice.count(PROOF_FRAMING_CONTEXT), 1) + + def test_append_proof_framing_defaults_legacy_instance_to_enabled(self): + coordinator = self._gate_coordinator() + del coordinator._allow_mathematical_proofs + coordinator._proof_framing_active = True + coordinator._proof_framing_context = PROOF_FRAMING_CONTEXT + + result = coordinator._append_proof_framing("Research goal.") + + self.assertEqual(result, f"Research goal.\n\n{PROOF_FRAMING_CONTEXT}") + + def test_append_proof_framing_preserves_explicit_false_suppression(self): + coordinator = self._gate_coordinator() + coordinator._allow_mathematical_proofs = False + coordinator._proof_framing_active = True + coordinator._proof_framing_context = PROOF_FRAMING_CONTEXT + + result = coordinator._append_proof_framing("Research goal.") + + self.assertEqual(result, "Research goal.") + + def test_apply_proof_context_frames_before_library_injection(self): + class CapturingProofDatabase: + def __init__(self): + self.received = None + + def inject_into_prompt(self, prompt): + self.received = prompt + return f"LIBRARY::{prompt}" + + fake_database = CapturingProofDatabase() + coordinator_module.proof_database = fake_database + coordinator = self._gate_coordinator() + coordinator._proof_framing_active = True + coordinator._proof_framing_context = PROOF_FRAMING_CONTEXT + + result = coordinator._apply_proof_context("Research goal.") + + self.assertEqual( + fake_database.received, + f"Research goal.\n\n{PROOF_FRAMING_CONTEXT}", + ) + self.assertEqual(result, f"LIBRARY::{fake_database.received}") + + async def test_resume_restores_proof_framing_without_rerunning_gate(self): + fake_metadata = coordinator_module.research_metadata + fake_metadata.interrupted_workflow = True + fake_metadata.base_user_prompt = "Persisted base prompt." + fake_metadata.workflow_states.append( + { + "current_tier": "tier1_aggregation", + "proof_framing_active": True, + "proof_framing_context": PROOF_FRAMING_CONTEXT, + "proof_framing_reasoning": "Persisted positive decision.", + } + ) + fake_api = _FakeAPIClientManager( + error=AssertionError("resume must not rerun proof framing gate") + ) + coordinator_module.api_client_manager = fake_api + coordinator = self._gate_coordinator() + + async def identity_normalize(state): + return state + + coordinator._normalize_resume_state = identity_normalize + await coordinator._check_resume_state() + + self.assertEqual(fake_api.calls, []) + self.assertTrue(coordinator._proof_framing_active) + self.assertEqual(coordinator._proof_framing_context, PROOF_FRAMING_CONTEXT) + self.assertEqual( + coordinator._proof_framing_reasoning, + "Persisted positive decision.", + ) + self.assertEqual(coordinator._user_research_prompt, "Persisted base prompt.") + + class AutonomousProofRoundTests(unittest.IsolatedAsyncioTestCase): def setUp(self): self.old_lean4_enabled = system_config.lean4_enabled @@ -415,6 +772,36 @@ async def test_later_round_resume_does_not_overwrite_active_checkpoint(self): ) self.assertNotEqual(fake_metadata.mark_calls[0], "automatic") + async def test_deferred_overflow_resumes_without_replaying_completed_candidate(self): + fake_stage = _DeferredThenCompleteProofStage() + coordinator = _configured_coordinator(fake_stage) + fake_metadata = coordinator_module.research_metadata + + first_status = await coordinator._run_proof_verification( + content="Paper source.", + source_type="paper", + source_id="paper_deferred", + source_title="Paper title", + ) + + self.assertEqual(first_status, "complete") + self.assertEqual(fake_metadata.mark_calls, []) + self.assertEqual(fake_metadata.checkpoint["status"], "deferred") + self.assertEqual(fake_metadata.checkpoint["processed_candidate_ids"], ["done"]) + + second_status = await coordinator._run_proof_verification( + content="Paper source.", + source_type="paper", + source_id="paper_deferred", + source_title="Paper title", + ) + + self.assertEqual(second_status, "complete") + self.assertEqual(len(fake_stage.calls), 2) + resumed = fake_stage.calls[1]["theorem_candidates"] + self.assertEqual([candidate.theorem_id for candidate in resumed], ["deferred"]) + self.assertEqual(fake_metadata.mark_calls, ["automatic"]) + async def test_automatic_follow_up_rounds_hold_source_reservation(self): class ReservationCheckingStage(_FakeProofStage): def __init__(self, totals): diff --git a/tests/unit/test_autonomous_strict_response_contracts.py b/tests/unit/test_autonomous_strict_response_contracts.py new file mode 100644 index 0000000..7fdcd09 --- /dev/null +++ b/tests/unit/test_autonomous_strict_response_contracts.py @@ -0,0 +1,193 @@ +import json + +import pytest + +from backend.autonomous.core.autonomous_coordinator import AutonomousCoordinator # noqa: F401 +from backend.autonomous.agents.final_answer.answer_format_selector import ( + AnswerFormatSelector, +) +from backend.autonomous.agents.final_answer.volume_organizer import VolumeOrganizer +from backend.autonomous.agents.paper_title_selector import PaperTitleSelectorAgent +from backend.autonomous.agents.topic_selector import TopicSelectorAgent +from backend.shared.api_client_manager import api_client_manager +from backend.shared.models import CertaintyAssessment + + +def _response(payload: dict) -> dict: + return {"choices": [{"message": {"content": json.dumps(payload)}}]} + + +@pytest.fixture +def completion_stub(monkeypatch): + async def install(payload: dict) -> None: + async def fake_generate_completion(**_kwargs): + return _response(payload) + + async def fake_prewarm(**_kwargs): + return None + + monkeypatch.setattr( + api_client_manager, "generate_completion", fake_generate_completion + ) + monkeypatch.setattr( + api_client_manager, "prewarm_assistant_memory_context", fake_prewarm + ) + + return install + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "payload", + ( + {"reasoning": "Missing format"}, + {"answer_format": "brief", "reasoning": "Invalid enum"}, + {"answer_format": "short_form", "reasoning": " "}, + ), +) +async def test_answer_format_rejects_missing_invalid_or_blank_required_fields( + completion_stub, payload +) -> None: + await completion_stub(payload) + selector = AnswerFormatSelector("submitter", "validator", 8192, 1024) + + result = await selector._generate_selection( + "Exact objective", + CertaintyAssessment( + certainty_level="partial_answer", + known_certainties_summary="Some support", + reasoning="Evidence is incomplete", + ), + [{"paper_id": "p1", "title": "Paper"}], + ) + + assert result is None + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "payload", + ( + { + "volume_title": "Volume", + "chapters": [], + "outline_complete": True, + "reasoning": "No chapters", + }, + { + "volume_title": "Volume", + "chapters": [ + { + "chapter_type": "existing_paper", + "paper_id": "p1", + "title": "Body", + "order": 1, + } + ], + "outline_complete": "true", + "reasoning": "Wrong boolean type", + }, + { + "volume_title": "Volume", + "chapters": [ + { + "chapter_type": "introduction", + "title": "Introduction", + "order": 1, + }, + { + "chapter_type": "existing_paper", + "paper_id": "p1", + "title": "Body", + "order": 2, + }, + ], + "outline_complete": True, + "reasoning": "Missing conclusion", + }, + ), +) +async def test_volume_organizer_rejects_incomplete_or_repaired_contracts( + completion_stub, payload +) -> None: + await completion_stub(payload) + organizer = VolumeOrganizer("submitter", "validator", 8192, 1024) + + result = await organizer._generate_organization( + "Exact objective", + CertaintyAssessment( + certainty_level="partial_answer", + known_certainties_summary="Some support", + reasoning="Evidence is incomplete", + ), + [{"paper_id": "p1", "title": "Paper"}], + ) + + assert result is None + + +@pytest.mark.asyncio +async def test_volume_organizer_does_not_force_completion_at_iteration_limit( + monkeypatch, +) -> None: + organizer = VolumeOrganizer("submitter", "validator", 8192, 1024) + organizer.MAX_ITERATIONS = 1 + + async def fake_generate(*_args, **_kwargs): + from backend.shared.models import VolumeChapter, VolumeOrganization + + return VolumeOrganization( + volume_title="Incomplete", + chapters=[ + VolumeChapter( + chapter_type="introduction", title="Introduction", order=1 + ), + VolumeChapter( + chapter_type="conclusion", title="Conclusion", order=2 + ), + ], + outline_complete=False, + revision_reasoning="Needs another pass", + ) + + async def fake_validate(*_args, **_kwargs): + return True, "Valid but incomplete" + + monkeypatch.setattr(organizer, "_generate_organization", fake_generate) + monkeypatch.setattr(organizer, "_validate_organization", fake_validate) + + result = await organizer.organize_volume( + "Exact objective", + CertaintyAssessment( + certainty_level="partial_answer", + known_certainties_summary="Some support", + reasoning="Evidence is incomplete", + ), + [{"paper_id": "p1", "title": "Paper"}], + ) + + assert result is None + + +@pytest.mark.asyncio +async def test_topic_selector_requires_non_empty_reasoning(completion_stub) -> None: + await completion_stub( + {"action": "new_topic", "topic_prompt": "Direct route", "reasoning": ""} + ) + selector = TopicSelectorAgent("model", 8192, 1024) + + result = await selector.select_topic("Exact objective", [], [], "Candidate") + + assert result is None + + +@pytest.mark.asyncio +async def test_paper_title_requires_non_empty_reasoning(completion_stub) -> None: + await completion_stub({"paper_title": "Strong title", "reasoning": ""}) + selector = PaperTitleSelectorAgent("model", "validator", 8192, 1024) + + result = await selector._generate_title( + "Exact objective", "Topic", "Summary", [], [], "", "" + ) + + assert result is None diff --git a/tests/test_build_info.py b/tests/unit/test_build_info.py similarity index 75% rename from tests/test_build_info.py rename to tests/unit/test_build_info.py index 5cbd7a8..ea46f5c 100644 --- a/tests/test_build_info.py +++ b/tests/unit/test_build_info.py @@ -4,6 +4,7 @@ from unittest import TestCase, main, mock from backend.shared import build_info +import moto_updater class BuildInfoArchiveIdentityTests(TestCase): @@ -43,6 +44,21 @@ def test_get_build_info_uses_archival_head_for_zip_archives(self) -> None: self.assertEqual(resolved.build_commit, "0123456789abcdef0123456789abcdef01234567") + def test_default_and_manifest_contract_versions_match(self) -> None: + manifest = json.loads(build_info.BUILD_MANIFEST_PATH.read_text(encoding="utf-8")) + self.assertEqual( + build_info._DEFAULT_BUILD_INFO["api_contract_version"], + "build5-v73", + ) + self.assertEqual( + manifest["api_contract_version"], + build_info._DEFAULT_BUILD_INFO["api_contract_version"], + ) + self.assertEqual( + moto_updater._DEFAULT_MANIFEST["api_contract_version"], + build_info._DEFAULT_BUILD_INFO["api_contract_version"], + ) + if __name__ == "__main__": main() diff --git a/tests/unit/test_chroma_cache_cleanup.py b/tests/unit/test_chroma_cache_cleanup.py new file mode 100644 index 0000000..bb95181 --- /dev/null +++ b/tests/unit/test_chroma_cache_cleanup.py @@ -0,0 +1,141 @@ +import sqlite3 +import tempfile +import unittest +from pathlib import Path + +from backend.aggregator.core.chroma_cache import ( + complete_chroma_cache_rebuild, + maintain_chroma_cache_directory, + quarantine_chroma_cache, + recover_interrupted_chroma_rebuild, +) + + +def _create_chroma_sqlite(path: Path, referenced_ids: list[str]) -> None: + con = sqlite3.connect(path) + try: + cur = con.cursor() + cur.execute("create table collections (id text, name text)") + cur.execute("create table segments (id text, type text, scope text, collection text)") + for index, referenced_id in enumerate(referenced_ids): + cur.execute( + "insert into segments values (?, ?, ?, ?)", + (referenced_id, "urn:chroma:segment/vector/hnsw-local-persisted", "VECTOR", f"collection-{index}"), + ) + con.commit() + finally: + con.close() + + +class ChromaCacheCleanupTests(unittest.TestCase): + def test_removes_only_chroma_cache_when_orphan_uuid_buildup_detected(self) -> None: + with tempfile.TemporaryDirectory() as temp_dir: + data_root = Path(temp_dir) + chroma_dir = data_root / "chroma_db" + chroma_dir.mkdir() + + durable_file = data_root / "rag_shared_training.txt" + durable_file.write_text("keep me", encoding="utf-8") + durable_session = data_root / "auto_sessions" / "session_1" + durable_session.mkdir(parents=True) + (durable_session / "paper.txt").write_text("keep session", encoding="utf-8") + + _create_chroma_sqlite(chroma_dir / "chroma.sqlite3", referenced_ids=[]) + for i in range(70): + orphan_dir = chroma_dir / f"00000000-0000-0000-0000-{i:012x}" + orphan_dir.mkdir() + (orphan_dir / "index.bin").write_text("cache", encoding="utf-8") + (chroma_dir / "not-a-uuid.txt").write_text("cache", encoding="utf-8") + + result = maintain_chroma_cache_directory(chroma_dir, data_root) + + self.assertTrue(result.reset_performed) + self.assertEqual(result.unreferenced_uuid_dir_count, 70) + self.assertTrue(chroma_dir.exists()) + self.assertEqual(list(chroma_dir.iterdir()), []) + self.assertEqual(durable_file.read_text(encoding="utf-8"), "keep me") + self.assertEqual((durable_session / "paper.txt").read_text(encoding="utf-8"), "keep session") + + def test_skips_cleanup_below_threshold(self) -> None: + with tempfile.TemporaryDirectory() as temp_dir: + data_root = Path(temp_dir) + chroma_dir = data_root / "chroma_db" + chroma_dir.mkdir() + _create_chroma_sqlite(chroma_dir / "chroma.sqlite3", referenced_ids=[]) + orphan_dir = chroma_dir / "00000000-0000-0000-0000-000000000001" + orphan_dir.mkdir() + + result = maintain_chroma_cache_directory(chroma_dir, data_root) + + self.assertFalse(result.reset_performed) + self.assertEqual(result.reason, "below_threshold") + self.assertTrue(orphan_dir.exists()) + + def test_rejects_chroma_path_outside_data_root(self) -> None: + with tempfile.TemporaryDirectory() as temp_dir: + root = Path(temp_dir) + data_root = root / "data" + outside = root / "outside_chroma" + data_root.mkdir() + outside.mkdir() + + with self.assertRaises(ValueError): + maintain_chroma_cache_directory(outside, data_root) + + def test_skips_cleanup_when_metadata_is_missing(self) -> None: + with tempfile.TemporaryDirectory() as temp_dir: + data_root = Path(temp_dir) + chroma_dir = data_root / "chroma_db" + chroma_dir.mkdir() + for i in range(70): + orphan_dir = chroma_dir / f"00000000-0000-0000-0000-{i:012x}" + orphan_dir.mkdir() + + result = maintain_chroma_cache_directory(chroma_dir, data_root) + + self.assertFalse(result.reset_performed) + self.assertEqual(result.reason, "missing_sqlite_metadata") + self.assertEqual(len([p for p in chroma_dir.iterdir() if p.is_dir()]), 70) + + def test_rejects_data_root_as_chroma_path(self) -> None: + with tempfile.TemporaryDirectory() as temp_dir: + data_root = Path(temp_dir) + + with self.assertRaises(ValueError): + maintain_chroma_cache_directory(data_root, data_root) + + def test_quarantine_rebuild_preserves_durable_files(self) -> None: + with tempfile.TemporaryDirectory() as temp_dir: + data_root = Path(temp_dir) + chroma_dir = data_root / "chroma_db" + chroma_dir.mkdir() + (chroma_dir / "chroma.sqlite3").write_text("cache", encoding="utf-8") + durable = data_root / "rag_shared_training.txt" + durable.write_text("durable", encoding="utf-8") + + fresh, quarantine = quarantine_chroma_cache(chroma_dir, data_root) + self.assertEqual(list(fresh.iterdir()), []) + self.assertIsNotNone(quarantine) + self.assertTrue((quarantine / "chroma.sqlite3").exists()) + complete_chroma_cache_rebuild(fresh, data_root, quarantine) + + self.assertFalse(quarantine.exists()) + self.assertEqual(durable.read_text(encoding="utf-8"), "durable") + + def test_interrupted_rebuild_restores_last_cache(self) -> None: + with tempfile.TemporaryDirectory() as temp_dir: + data_root = Path(temp_dir) + chroma_dir = data_root / "chroma_db" + chroma_dir.mkdir() + (chroma_dir / "old.cache").write_text("old", encoding="utf-8") + + fresh, quarantine = quarantine_chroma_cache(chroma_dir, data_root) + fresh.rmdir() + recover_interrupted_chroma_rebuild(chroma_dir, data_root) + + self.assertTrue((chroma_dir / "old.cache").exists()) + self.assertFalse(quarantine.exists()) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_compiler_marker_visibility.py b/tests/unit/test_compiler_marker_visibility.py similarity index 100% rename from tests/test_compiler_marker_visibility.py rename to tests/unit/test_compiler_marker_visibility.py diff --git a/tests/unit/test_compiler_validator_json_contract.py b/tests/unit/test_compiler_validator_json_contract.py new file mode 100644 index 0000000..04c3c10 --- /dev/null +++ b/tests/unit/test_compiler_validator_json_contract.py @@ -0,0 +1,321 @@ +import json + +import pytest + +from backend.compiler.validation import compiler_validator as validator_module +from backend.compiler.validation.compiler_validator import ( + CompilerValidator, + ValidatorContractError, +) +from backend.shared.config import system_config +from backend.shared.models import CompilerSubmission + + +def _valid_response(**overrides): + data = { + "decision": "accept", + "reasoning": "The submission satisfies the applicable criteria.", + "coherence_check": True, + "rigor_check": True, + "placement_check": True, + } + data.update(overrides) + return data + + +@pytest.mark.parametrize( + "payload", + [ + {"reasoning": "Missing decision", "coherence_check": True, "rigor_check": True, "placement_check": True}, + _valid_response(decision="maybe"), + _valid_response(reasoning=" "), + _valid_response(coherence_check=1), + _valid_response(rigor_check="true"), + _valid_response(placement_check=None), + ], +) +def test_primary_validation_contract_rejects_invalid_shapes(payload) -> None: + with pytest.raises(ValidatorContractError): + CompilerValidator._validate_primary_validation_contract( + payload, + require_checks=True, + ) + + +def test_optional_solution_path_extension_does_not_invalidate_primary_result() -> None: + payload = _valid_response(solution_path_update="malformed optional extension") + + parsed = CompilerValidator._validate_primary_validation_contract( + payload, + require_checks=True, + ) + + assert parsed["decision"] == "accept" + assert parsed["solution_path_update"] == "malformed optional extension" + + +@pytest.mark.asyncio +async def test_malformed_accept_prose_cannot_be_accepted_after_bounded_retry( + monkeypatch: pytest.MonkeyPatch, +) -> None: + validator = CompilerValidator(model_name="test-model", user_prompt="Solve it.") + calls = [] + monkeypatch.setattr(system_config, "compiler_validator_context_window", 8000) + monkeypatch.setattr(system_config, "compiler_validator_max_output_tokens", 1000) + + async def fake_generate_completion(**kwargs): + calls.append(kwargs) + return { + "choices": [ + {"message": {"content": "decision: accept because it looks valid"}} + ] + } + + monkeypatch.setattr( + validator_module.api_client_manager, + "generate_completion", + fake_generate_completion, + ) + + with pytest.raises(ValueError): + await validator._parse_json_with_retry( + "decision: accept", + "ORIGINAL PROMPT", + "", + ) + + assert len(calls) == 1 + + +@pytest.mark.asyncio +async def test_retry_uses_sanitized_failed_output_and_succeeds_once( + monkeypatch: pytest.MonkeyPatch, +) -> None: + validator = CompilerValidator(model_name="test-model", user_prompt="Solve it.") + captured = {} + monkeypatch.setattr(system_config, "compiler_validator_context_window", 8000) + monkeypatch.setattr(system_config, "compiler_validator_max_output_tokens", 1000) + + async def fake_generate_completion(**kwargs): + captured.update(kwargs) + return { + "choices": [ + {"message": {"content": json.dumps(_valid_response())}} + ] + } + + monkeypatch.setattr( + validator_module.api_client_manager, + "generate_completion", + fake_generate_completion, + ) + + parsed = await validator._parse_json_with_retry( + "<think>private reasoning</think> decision: accept", + "ORIGINAL PROMPT", + "", + ) + + assert parsed["decision"] == "accept" + messages = captured["messages"] + assert len(messages) == 3 + assert "<think>" not in messages[1]["content"] + assert "private reasoning" not in messages[1]["content"] + + +@pytest.mark.asyncio +async def test_retry_provider_value_error_is_not_misclassified_as_contract_failure( + monkeypatch: pytest.MonkeyPatch, +) -> None: + validator = CompilerValidator(model_name="test-model", user_prompt="Solve it.") + monkeypatch.setattr(system_config, "compiler_validator_context_window", 8000) + monkeypatch.setattr(system_config, "compiler_validator_max_output_tokens", 1000) + + async def fake_generate_completion(**kwargs): + raise ValueError("provider configuration is invalid") + + monkeypatch.setattr( + validator_module.api_client_manager, + "generate_completion", + fake_generate_completion, + ) + + with pytest.raises(ValueError, match="provider configuration"): + await validator._parse_json_with_retry( + "decision: accept", + "ORIGINAL PROMPT", + "", + ) + + +@pytest.mark.asyncio +async def test_brainstorm_contract_requires_decision_and_reasoning_only( + monkeypatch: pytest.MonkeyPatch, +) -> None: + validator = CompilerValidator(model_name="test-model", user_prompt="Solve it.") + + parsed = await validator._parse_json_with_retry( + json.dumps({"decision": "reject", "reasoning": "The operation is unsupported."}), + "ORIGINAL PROMPT", + "", + require_checks=False, + ) + + assert parsed == { + "decision": "reject", + "reasoning": "The operation is unsupported.", + } + + +@pytest.mark.asyncio +async def test_lean_placement_does_not_force_rigor_before_structure_is_valid( + monkeypatch: pytest.MonkeyPatch, +) -> None: + validator = CompilerValidator(model_name="test-model", user_prompt="Write it.") + monkeypatch.setattr(system_config, "compiler_validator_context_window", 8000) + monkeypatch.setattr(system_config, "compiler_validator_max_output_tokens", 1000) + + async def fake_ensure_markers_intact(): + return False + + responses = iter( + [ + {"choices": [{"message": {"content": json.dumps({ + "decision": "accept", + "reasoning": "Looks placed.", + "coherence_check": True, + "rigor_check": "false", + "placement_check": True, + })}}]}, + {"choices": [{"message": {"content": "still malformed"}}]}, + ] + ) + + async def fake_generate_completion(**kwargs): + return next(responses) + + monkeypatch.setattr( + validator_module.paper_memory, + "ensure_markers_intact", + fake_ensure_markers_intact, + ) + monkeypatch.setattr( + validator_module.api_client_manager, + "generate_completion", + fake_generate_completion, + ) + + submission = CompilerSubmission( + submission_id="lean-invalid-structure", + mode="rigor", + operation="insert_after", + old_string="Anchor.", + new_string="Verified theorem.", + content="Verified theorem.", + reasoning="Place it.", + metadata={"rigor_mode": "lean_placement"}, + ) + result = await validator.validate_submission( + submission, + current_paper="Anchor.", + current_outline="I. Introduction\nII. Body\nIII. Conclusion", + ) + + assert result.decision == "reject" + assert result.json_valid is False + assert result.rigor_check is False + assert result.summary == "Invalid validator JSON contract" + + +def test_full_content_is_rejected_for_non_empty_paper() -> None: + validator = CompilerValidator(model_name="test-model", user_prompt="Write it.") + submission = CompilerSubmission( + submission_id="destructive-full-content", + mode="construction", + operation="full_content", + old_string="", + new_string="Replacement paper.", + content="Replacement paper.", + reasoning="Replace everything.", + ) + + result = validator._pre_validate_exact_string_match( + submission, + current_paper="Existing paper content.", + current_outline="I. Introduction\nII. Body\nIII. Conclusion", + ) + + assert result is not None + assert result.decision == "reject" + assert "NON_EMPTY_FULL_CONTENT_ERROR" in result.reasoning + assert result.placement_check is False + + +def test_full_content_remains_valid_for_empty_paper() -> None: + validator = CompilerValidator(model_name="test-model", user_prompt="Write it.") + submission = CompilerSubmission( + submission_id="initial-full-content", + mode="construction", + operation="full_content", + old_string="", + new_string="Initial body.", + content="Initial body.", + reasoning="Start the paper.", + ) + + result = validator._pre_validate_exact_string_match( + submission, + current_paper="", + current_outline="I. Introduction\nII. Body\nIII. Conclusion", + ) + + assert result is None + + +@pytest.mark.asyncio +@pytest.mark.parametrize("failed_field", ["coherence_check", "rigor_check", "placement_check"]) +async def test_acceptance_with_failed_check_is_converted_to_rejection( + monkeypatch: pytest.MonkeyPatch, + failed_field: str, +) -> None: + validator = CompilerValidator(model_name="test-model", user_prompt="Write it.") + monkeypatch.setattr(system_config, "compiler_validator_context_window", 8000) + monkeypatch.setattr(system_config, "compiler_validator_max_output_tokens", 1000) + + async def fake_ensure_markers_intact(): + return False + + payload = _valid_response(**{failed_field: False}) + + async def fake_generate_completion(**kwargs): + return {"choices": [{"message": {"content": json.dumps(payload)}}]} + + monkeypatch.setattr( + validator_module.paper_memory, + "ensure_markers_intact", + fake_ensure_markers_intact, + ) + monkeypatch.setattr( + validator_module.api_client_manager, + "generate_completion", + fake_generate_completion, + ) + + submission = CompilerSubmission( + submission_id=f"inconsistent-{failed_field}", + mode="construction", + operation="insert_after", + old_string="Anchor.", + new_string="New material.", + content="New material.", + reasoning="Add material.", + ) + result = await validator.validate_submission( + submission, + current_paper="Anchor.", + current_outline="I. Introduction\nII. Body\nIII. Conclusion", + ) + + assert result.decision == "reject" + assert failed_field in result.reasoning + assert result.json_valid is True diff --git a/tests/test_compiler_wolfram_tracking.py b/tests/unit/test_compiler_wolfram_tracking.py similarity index 100% rename from tests/test_compiler_wolfram_tracking.py rename to tests/unit/test_compiler_wolfram_tracking.py diff --git a/tests/unit/test_connectivity_status.py b/tests/unit/test_connectivity_status.py new file mode 100644 index 0000000..5f36a5e --- /dev/null +++ b/tests/unit/test_connectivity_status.py @@ -0,0 +1,111 @@ +from unittest import mock + +import pytest + +from backend.api.routes import connectivity +from backend.api.routes.connectivity import ( + ConnectivityToggleRequest, + _syntheticlib4_is_outdated, +) +from backend.shared.config import system_config + + +def test_syntheticlib4_outdated_when_cached_snapshot_has_expired_membership() -> None: + assert _syntheticlib4_is_outdated( + { + "credential_configured": True, + "auth_mode": "api_key", + "authenticated": True, + "membership_active": False, + }, + ready=True, + ) + + +def test_syntheticlib4_inactive_runtime_is_not_outdated_without_a_ready_snapshot() -> None: + assert not _syntheticlib4_is_outdated( + { + "credential_configured": False, + "auth_mode": "inactive", + "authenticated": False, + "membership_active": False, + }, + ready=False, + ) + + +def test_syntheticlib4_local_snapshot_with_explicit_membership_failure_is_outdated() -> None: + assert _syntheticlib4_is_outdated( + { + "credential_configured": False, + "auth_mode": "local_snapshot", + "authenticated": True, + "membership_active": False, + }, + ready=True, + ) + + +def test_syntheticlib4_expired_access_timestamp_is_outdated() -> None: + assert _syntheticlib4_is_outdated( + { + "credential_configured": True, + "auth_mode": "api_key", + "authenticated": True, + "membership_active": True, + "access_expires_at": "2000-01-01T00:00:00Z", + }, + ready=True, + ) + + +def test_syntheticlib4_future_access_timestamp_is_not_outdated() -> None: + assert not _syntheticlib4_is_outdated( + { + "credential_configured": True, + "auth_mode": "api_key", + "authenticated": True, + "membership_active": True, + "access_expires_at": "2999-01-01T00:00:00Z", + }, + ready=True, + ) + + +@pytest.mark.asyncio +async def test_disabling_session_history_clears_live_and_durable_assistant_state() -> None: + old_enabled = system_config.agent_conversation_memory_enabled + try: + with ( + mock.patch.object(connectivity, "_workflow_is_active", return_value=False), + mock.patch.object(connectivity, "save_connectivity_runtime_settings"), + mock.patch.object( + connectivity.assistant_proof_search_coordinator, + "stop_all", + new=mock.AsyncMock(), + ) as stop_all, + mock.patch.object( + connectivity.assistant_proof_search_coordinator, + "clear_cooldown_state", + new=mock.AsyncMock(), + ) as clear_cooldown_state, + mock.patch.object( + connectivity, + "get_connectivity_status", + new=mock.AsyncMock(return_value={"success": True}), + ), + ): + result = await connectivity.update_connectivity_toggles( + ConnectivityToggleRequest(agent_conversation_memory_enabled=False) + ) + finally: + system_config.agent_conversation_memory_enabled = old_enabled + + assert result == {"success": True} + stop_all.assert_awaited_once_with( + clear_packs=True, + broadcast=True, + reason="agent_conversation_memory_disabled", + ) + clear_cooldown_state.assert_awaited_once_with() + diff --git a/tests/test_creativity_emphasis_boost.py b/tests/unit/test_creativity_emphasis_boost.py similarity index 88% rename from tests/test_creativity_emphasis_boost.py rename to tests/unit/test_creativity_emphasis_boost.py index 41a58da..f6c5127 100644 --- a/tests/test_creativity_emphasis_boost.py +++ b/tests/unit/test_creativity_emphasis_boost.py @@ -57,13 +57,18 @@ def test_aggregator_prompt_hides_lean_proof_schema_when_disabled(self) -> None: self.assertIn("lean_proof", enabled_prompt) self.assertIn("Lean proof candidate", enabled_prompt) - def test_brainstorm_submitter_prompts_keep_novel_insight_task_line(self) -> None: + def test_brainstorm_submitter_prompts_keep_direct_novelty_pressure(self) -> None: aggregator_prompt = build_submitter_prompt("user", "context") leanoj_prompt = build_brainstorm_prompt("prompt", "template", "topic", [], [], []) - expected_line = "Generate a novel mathematical insight that advances the user's goal." - self.assertIn(f"YOUR TASK:\n{expected_line}\nGenerate the strongest rigorous mathematical contribution", aggregator_prompt) - self.assertIn(f"YOUR TASK:\n{expected_line}\nGenerate one concrete idea", leanoj_prompt) + self.assertIn( + "YOUR TASK:\nAggressively pursue the strongest credible and genuinely novel solution " + "to the user's exact objective.", + aggregator_prompt, + ) + self.assertIn("WHOLE question", aggregator_prompt) + expected_leanoj_line = "Generate a novel mathematical insight that advances the user's goal." + self.assertIn(f"YOUR TASK:\n{expected_leanoj_line}\nGenerate one concrete idea", leanoj_prompt) def test_aggregator_submitter_uses_every_fifth_valid_submission_slot(self) -> None: submitter = SubmitterAgent( diff --git a/tests/unit/test_free_model_manager.py b/tests/unit/test_free_model_manager.py new file mode 100644 index 0000000..4ef84e5 --- /dev/null +++ b/tests/unit/test_free_model_manager.py @@ -0,0 +1,145 @@ +from unittest import IsolatedAsyncioTestCase, mock + +from backend.shared.free_model_manager import FreeModelManager, supports_text_chat_model +from backend.shared.openrouter_client import OpenRouterClient + + +def _free_model(model_id, *, architecture=None, context_length=1000): + return { + "id": model_id, + "context_length": context_length, + "pricing": { + "prompt": "0", + "completion": "0", + }, + "architecture": architecture or {}, + } + + +def test_free_model_rotation_caches_only_text_chat_models(): + manager = FreeModelManager() + manager.update_cached_models( + [ + _free_model( + "google/lyria-clip-preview:free", + architecture={ + "input_modalities": ["text"], + "output_modalities": ["image"], + }, + context_length=999999, + ), + _free_model( + "image-only/free:free", + architecture={ + "input_modalities": ["image"], + "output_modalities": ["text"], + }, + context_length=500000, + ), + _free_model( + "openrouter/text-model:free", + architecture={ + "input_modalities": ["text"], + "output_modalities": ["text"], + }, + context_length=1000, + ), + _free_model( + "missing-modality/free:free", + architecture={}, + context_length=750000, + ), + ] + ) + + assert ( + manager.get_alternative_free_model("exhausted/model:free") + == "openrouter/text-model:free" + ) + + +def test_text_chat_filter_requires_modality_metadata(): + assert supports_text_chat_model( + _free_model("chat/free:free", architecture={"modality": "text->text"}) + ) + assert not supports_text_chat_model(_free_model("missing-modality/free:free")) + + +def test_free_model_rotation_understands_compact_modality_string(): + manager = FreeModelManager() + manager.update_cached_models( + [ + _free_model( + "image-generator/free:free", + architecture={"modality": "text->image"}, + context_length=999999, + ), + _free_model( + "chat/free:free", + architecture={"modality": "text->text"}, + context_length=1000, + ), + ] + ) + + assert manager.get_alternative_free_model("exhausted/model:free") == "chat/free:free" + + +def test_failed_rotated_model_is_skipped_next_time(): + manager = FreeModelManager() + manager.update_cached_models( + [ + _free_model( + "bad/free:free", + architecture={"modality": "text->text"}, + context_length=2000, + ), + _free_model( + "good/free:free", + architecture={"modality": "text->text"}, + context_length=1000, + ), + ] + ) + + assert manager.get_alternative_free_model("original/free:free") == "bad/free:free" + manager.mark_model_failed("bad/free:free") + assert manager.get_alternative_free_model("original/free:free") == "good/free:free" + + +class OpenRouterFreeModelListTests(IsolatedAsyncioTestCase): + async def test_free_only_list_returns_only_text_chat_models(self): + client = OpenRouterClient("test-key") + response = mock.Mock() + response.raise_for_status.return_value = None + response.json.return_value = { + "data": [ + _free_model( + "google/lyria-clip-preview:free", + architecture={"modality": "text->image"}, + ), + _free_model( + "speech/free:free", + architecture={ + "input_modalities": ["audio"], + "output_modalities": ["text"], + }, + ), + _free_model( + "chat/free:free", + architecture={"modality": "text->text"}, + ), + _free_model("missing-modality/free:free"), + { + "id": "paid/chat", + "pricing": {"prompt": "0.1", "completion": "0.1"}, + "architecture": {"modality": "text->text"}, + }, + ] + } + + with mock.patch.object(client.client, "get", mock.AsyncMock(return_value=response)): + models = await client.list_models(free_only=True, raise_on_error=True) + + await client.close() + assert [model["id"] for model in models] == ["chat/free:free"] diff --git a/tests/test_lean4_client.py b/tests/unit/test_lean4_client.py similarity index 100% rename from tests/test_lean4_client.py rename to tests/unit/test_lean4_client.py diff --git a/tests/test_lm_studio_readiness.py b/tests/unit/test_lm_studio_readiness.py similarity index 100% rename from tests/test_lm_studio_readiness.py rename to tests/unit/test_lm_studio_readiness.py diff --git a/tests/test_model_error_utils.py b/tests/unit/test_model_error_utils.py similarity index 100% rename from tests/test_model_error_utils.py rename to tests/unit/test_model_error_utils.py diff --git a/tests/test_paper_memory_appendix.py b/tests/unit/test_paper_memory_appendix.py similarity index 52% rename from tests/test_paper_memory_appendix.py rename to tests/unit/test_paper_memory_appendix.py index e9a67a0..ca75518 100644 --- a/tests/test_paper_memory_appendix.py +++ b/tests/unit/test_paper_memory_appendix.py @@ -5,6 +5,7 @@ from backend.compiler.memory.paper_memory import ( APPENDIX_EMPTY_PLACEHOLDER, ABSTRACT_PLACEHOLDER, + AI_SELF_REVIEW_SECTION_HEADER, CONCLUSION_PLACEHOLDER, INTRO_PLACEHOLDER, PAPER_ANCHOR, @@ -100,6 +101,89 @@ async def test_latex_conclusion_content_removes_stale_placeholder(self) -> None: finally: system_config.compiler_paper_file = old_paper_file + async def test_repairs_preserve_appendix_then_self_review_order(self) -> None: + for repair_method in ("ensure_placeholders_exist", "ensure_markers_intact"): + old_paper_file = system_config.compiler_paper_file + with tempfile.TemporaryDirectory() as tmpdir: + try: + system_config.compiler_paper_file = str( + Path(tmpdir) / f"{repair_method}.txt" + ) + memory = PaperMemory() + await memory.initialize() + theorem = ( + "### Verified Theorem\n" + "Proof ID: proof-order\n" + "```lean\n" + "theorem ordered : True := by trivial\n" + "```" + ) + self_review = ( + f"{AI_SELF_REVIEW_SECTION_HEADER}\n\n" + "The limitations section remains after verified proofs." + ) + abstract = "Abstract\n\n" + ("Substantive abstract content. " * 20) + await memory.update_paper( + f"{ABSTRACT_PLACEHOLDER}\n\n{abstract}\n\n" + f"{INTRO_PLACEHOLDER}\n\n" + "II. Body\n\nBody text.\n\n" + f"{CONCLUSION_PLACEHOLDER}\n\n" + f"{THEOREMS_APPENDIX_START}\n" + f"{theorem}\n" + f"{THEOREMS_APPENDIX_END}\n\n" + f"{self_review}\n" + ) + + self.assertTrue(await getattr(memory, repair_method)()) + paper = await memory.get_paper() + + self.assertEqual(paper.count("Proof ID: proof-order"), 1) + self.assertEqual( + paper.count(AI_SELF_REVIEW_SECTION_HEADER), 1 + ) + self.assertLess(paper.index("II. Body"), paper.index(theorem)) + self.assertLess(paper.index(theorem), paper.index(self_review)) + self.assertLess(paper.index(self_review), paper.index(PAPER_ANCHOR)) + finally: + system_config.compiler_paper_file = old_paper_file + + async def test_replacing_misordered_self_review_preserves_appendix(self) -> None: + old_paper_file = system_config.compiler_paper_file + with tempfile.TemporaryDirectory() as tmpdir: + try: + system_config.compiler_paper_file = str(Path(tmpdir) / "paper.txt") + memory = PaperMemory() + await memory.initialize() + theorem = ( + "Proof ID: proof-preserved\n" + "```lean\n" + "theorem preserved : True := by trivial\n" + "```" + ) + await memory.update_paper( + "II. Body\n\n" + f"{AI_SELF_REVIEW_SECTION_HEADER}\n\nOld review.\n\n" + f"{THEOREMS_APPENDIX_START}\n{theorem}\n" + f"{THEOREMS_APPENDIX_END}\n\n{PAPER_ANCHOR}" + ) + + self.assertTrue( + await memory.append_self_review_section("Replacement review.") + ) + paper = await memory.get_paper() + + self.assertEqual(paper.count("Proof ID: proof-preserved"), 1) + self.assertEqual(paper.count("theorem preserved : True"), 1) + self.assertEqual(paper.count(AI_SELF_REVIEW_SECTION_HEADER), 1) + self.assertIn("Replacement review.", paper) + self.assertNotIn("Old review.", paper) + self.assertLess( + paper.index(THEOREMS_APPENDIX_END), + paper.index(AI_SELF_REVIEW_SECTION_HEADER), + ) + finally: + system_config.compiler_paper_file = old_paper_file + if __name__ == "__main__": unittest.main() diff --git a/tests/test_proof_parallel_config.py b/tests/unit/test_proof_parallel_config.py similarity index 100% rename from tests/test_proof_parallel_config.py rename to tests/unit/test_proof_parallel_config.py diff --git a/tests/test_response_extraction.py b/tests/unit/test_response_extraction.py similarity index 100% rename from tests/test_response_extraction.py rename to tests/unit/test_response_extraction.py diff --git a/tests/test_rigor_lean_placement_validator.py b/tests/unit/test_rigor_lean_placement_validator.py similarity index 79% rename from tests/test_rigor_lean_placement_validator.py rename to tests/unit/test_rigor_lean_placement_validator.py index fb8ce39..f1aafab 100644 --- a/tests/test_rigor_lean_placement_validator.py +++ b/tests/unit/test_rigor_lean_placement_validator.py @@ -6,8 +6,13 @@ from backend.compiler.agents.high_param_submitter import HighParamSubmitter from backend.compiler.validation import compiler_validator as validator_module from backend.compiler.validation.compiler_validator import CompilerValidator +from backend.compiler.memory.paper_memory import ( + THEOREMS_APPENDIX_END, + THEOREMS_APPENDIX_START, +) from backend.shared.config import system_config from backend.shared.models import CompilerSubmission, ProofRecord +from backend.shared.solution_path import integration as solution_path_integration def _set_compiler_test_token_budgets(): @@ -54,6 +59,8 @@ async def test_lean_placement_prompt_and_forced_rigor_check(self) -> None: ) captured_prompt = {} + hook_calls = [] + enqueue_calls = [] async def fake_ensure_markers_intact() -> bool: return False @@ -71,6 +78,10 @@ async def fake_generate_completion(**kwargs): "coherence_check": True, "rigor_check": False, "placement_check": False, + "solution_path_update": { + "main_route": "Must be ignored for placement validation.", + "route": {"goal": "Must not enqueue."}, + }, } ) } @@ -78,11 +89,22 @@ async def fake_generate_completion(**kwargs): ] } + def fake_with_validator_hook(prompt, manager): + hook_calls.append((prompt, manager)) + return prompt + "\nsolution_path_update advisory proposal instruction" + + async def fake_enqueue_optional_update(*args, **kwargs): + enqueue_calls.append((args, kwargs)) + original_ensure = validator_module.paper_memory.ensure_markers_intact original_generate = validator_module.api_client_manager.generate_completion + original_hook = solution_path_integration.with_validator_hook + original_enqueue = solution_path_integration.enqueue_optional_update try: validator_module.paper_memory.ensure_markers_intact = fake_ensure_markers_intact validator_module.api_client_manager.generate_completion = fake_generate_completion + solution_path_integration.with_validator_hook = fake_with_validator_hook + solution_path_integration.enqueue_optional_update = fake_enqueue_optional_update result = await validator.validate_submission( submission, @@ -92,6 +114,8 @@ async def fake_generate_completion(**kwargs): finally: validator_module.paper_memory.ensure_markers_intact = original_ensure validator_module.api_client_manager.generate_completion = original_generate + solution_path_integration.with_validator_hook = original_hook + solution_path_integration.enqueue_optional_update = original_enqueue _restore_compiler_test_token_budgets(old_token_budgets) self.assertEqual(result.decision, "reject") @@ -99,6 +123,28 @@ async def fake_generate_completion(**kwargs): self.assertFalse(result.placement_check) self.assertIn("Lean 4 Verified Theorem Placement", captured_prompt["text"]) self.assertIn("MUST NOT re-evaluate", captured_prompt["text"]) + self.assertIn("LEAN 4 VERIFICATION CERTIFICATE", captured_prompt["text"]) + self.assertNotIn("solution_path_update", captured_prompt["text"]) + self.assertNotIn("advisory proposal instruction", captured_prompt["text"]) + self.assertEqual(hook_calls, []) + self.assertEqual(enqueue_calls, []) + + def test_ordinary_validator_prompt_exposes_theorem_appendix_boundaries(self) -> None: + validator = CompilerValidator(model_name="test-model", user_prompt="Write a paper.") + prompt = validator._get_paper_validation_system_prompt("construction") + + self.assertIn(THEOREMS_APPENDIX_START, prompt) + self.assertIn(THEOREMS_APPENDIX_END, prompt) + self.assertIn("crosses either theorem-appendix boundary", prompt) + self.assertIn("ordinary paper content inside the theorem appendix", prompt) + + def test_outline_prompt_does_not_require_optional_abstract(self) -> None: + validator = CompilerValidator(model_name="test-model", user_prompt="Write a paper.") + prompt = validator._get_outline_validation_system_prompt("outline_create") + + self.assertIn("Abstract is optional", prompt) + self.assertNotIn("MISSING_REQUIRED_SECTION - Abstract", prompt) + self.assertIn("MISSING_REQUIRED_SECTION - Introduction", prompt) class HighParamRigorProofRegistrationTests(unittest.IsolatedAsyncioTestCase): diff --git a/tests/unit/test_rigor_persistence_truthfulness.py b/tests/unit/test_rigor_persistence_truthfulness.py new file mode 100644 index 0000000..88632f3 --- /dev/null +++ b/tests/unit/test_rigor_persistence_truthfulness.py @@ -0,0 +1,96 @@ +from types import SimpleNamespace + +import pytest + +from backend.compiler.core.compiler_coordinator import CompilerCoordinator +from backend.compiler.memory.compiler_rejection_log import compiler_rejection_log +from backend.compiler.memory.paper_memory import paper_memory + + +def _lean_result(): + return SimpleNamespace( + proof_id="proof-1", + theorem_statement="theorem saved : True := by trivial", + lean_code="theorem saved : True := by trivial", + is_novel=True, + theorem_name="saved", + novelty_tier="novel", + placement_preference="appendix_only", + metadata={"placement_preference": "appendix_only"}, + initial_placement_submission=None, + ) + + +@pytest.mark.asyncio +async def test_appendix_failure_after_repair_is_not_reported_as_rigor_success( + monkeypatch, +) -> None: + coordinator = CompilerCoordinator() + events = [] + append_results = iter((False, False)) + + async def fake_append(_entry): + return next(append_results) + + async def fake_repair(): + return None + + async def fake_broadcast(event, payload): + events.append((event, payload)) + + monkeypatch.setattr(paper_memory, "append_to_theorems_appendix", fake_append) + monkeypatch.setattr(paper_memory, "ensure_markers_intact", fake_repair) + monkeypatch.setattr(coordinator, "_broadcast", fake_broadcast) + + result = await coordinator._place_or_appendix_fallback(_lean_result()) + + assert result is False + assert coordinator.rigor_acceptances == 0 + assert not any(event == "compiler_acceptance" for event, _ in events) + assert not any(event == "paper_updated" for event, _ in events) + assert any( + payload.get("placement_outcome") == "appendix_persistence_failed" + for event, payload in events + if event == "compiler_rejection" + ) + + +@pytest.mark.asyncio +async def test_appendix_success_after_marker_repair_is_reported_once( + monkeypatch, +) -> None: + coordinator = CompilerCoordinator() + events = [] + acceptance_ids = [] + append_results = iter((False, True)) + + async def fake_append(_entry): + return next(append_results) + + async def fake_repair(): + return None + + async def fake_word_count(): + return 42 + + async def fake_broadcast(event, payload): + events.append((event, payload)) + + async def fake_add_acceptance(submission_id, _mode, _preview): + acceptance_ids.append(submission_id) + + monkeypatch.setattr(paper_memory, "append_to_theorems_appendix", fake_append) + monkeypatch.setattr(paper_memory, "ensure_markers_intact", fake_repair) + monkeypatch.setattr(paper_memory, "get_word_count", fake_word_count) + monkeypatch.setattr(coordinator, "_broadcast", fake_broadcast) + monkeypatch.setattr( + compiler_rejection_log, "add_acceptance", fake_add_acceptance + ) + + result = await coordinator._place_or_appendix_fallback(_lean_result()) + + assert result is True + assert coordinator.rigor_acceptances == 1 + assert sum(event == "compiler_acceptance" for event, _ in events) == 1 + assert sum(event == "paper_updated" for event, _ in events) == 1 + assert acceptance_ids == ["rigor_appendix_requested_proof-1"] diff --git a/tests/test_rigor_prompt_source_context.py b/tests/unit/test_rigor_prompt_source_context.py similarity index 100% rename from tests/test_rigor_prompt_source_context.py rename to tests/unit/test_rigor_prompt_source_context.py diff --git a/tests/unit/test_run_all_tests.py b/tests/unit/test_run_all_tests.py new file mode 100644 index 0000000..b126b35 --- /dev/null +++ b/tests/unit/test_run_all_tests.py @@ -0,0 +1,67 @@ +from __future__ import annotations + +import pytest + +from tests.run_all_tests import ( + ROOT_DIR, + TESTS_DIR, + TestFileResult as FileRunResult, + discover_test_files, + parse_pytest_args, + resolve_requested_files, +) + + +def test_parse_pytest_args_preserves_quoted_expression() -> None: + assert parse_pytest_args('-k "workflow and not slow" --maxfail 1') == [ + "-k", + "workflow and not slow", + "--maxfail", + "1", + ] + + +def test_parse_pytest_args_accepts_empty_input() -> None: + assert parse_pytest_args("") == [] + + +def test_parse_pytest_args_preserves_windows_backslashes() -> None: + assert parse_pytest_args(r'--basetemp "C:\temp\pytest run"') == [ + "--basetemp", + r"C:\temp\pytest run", + ] + + +def test_pytest_no_match_exit_is_successful_skip() -> None: + result = FileRunResult( + path=TESTS_DIR / "test_example.py", + returncode=5, + elapsed_seconds=0.1, + stdout="", + stderr="", + allow_no_tests=True, + ) + + assert result.skipped + assert result.successful + assert not result.passed + + +def test_requested_file_must_be_a_pytest_file_under_tests() -> None: + with pytest.raises(SystemExit, match="Not a runnable pytest file"): + resolve_requested_files([str(ROOT_DIR / "package.json")]) + + +def test_discovery_includes_nested_test_packages() -> None: + discovered = {path.relative_to(ROOT_DIR).as_posix() for path in discover_test_files()} + + assert "tests/unit/test_build_info.py" in discovered + assert "tests/integration/test_allowed_outputs.py" in discovered + assert "tests/regressions/test_context_overflow_metadata.py" in discovered + assert "tests/workflow_scenarios/test_workflow_scenarios.py" in discovered + + +def test_requested_nested_test_file_is_resolved() -> None: + relative_path = "tests/integration/test_allowed_outputs.py" + + assert resolve_requested_files([relative_path]) == [(ROOT_DIR / relative_path).resolve()] diff --git a/tests/unit/test_smt_hint_invariance.py b/tests/unit/test_smt_hint_invariance.py new file mode 100644 index 0000000..53bbf03 --- /dev/null +++ b/tests/unit/test_smt_hint_invariance.py @@ -0,0 +1,171 @@ +from types import SimpleNamespace + +import pytest + +from backend.autonomous.core import proof_verification_stage as stage_module +from backend.autonomous.core.proof_verification_stage import ProofVerificationStage +from backend.shared.api_client_manager import RetryableProviderError +from backend.shared.config import system_config +from backend.shared.models import ProofAttemptFeedback, ProofCandidate, SmtHint + + +def _candidate() -> ProofCandidate: + return ProofCandidate( + theorem_id="t1", + statement="For n : Nat, n + 0 = n", + formal_sketch="Arithmetic equality over Nat", + ) + + +@pytest.mark.asyncio +async def test_disabled_smt_makes_no_translation_or_solver_call(monkeypatch) -> None: + stage = ProofVerificationStage() + agent = SimpleNamespace() + + async def forbidden_translate(**_kwargs): + raise AssertionError("translation must not run while SMT is disabled") + + agent.translate_candidate_to_smt = forbidden_translate + previous = system_config.smt_enabled + system_config.smt_enabled = False + try: + hint = await stage._run_smt_check( + user_prompt="Prove it", + source_type="brainstorm", + source_id="source", + base_event={}, + candidate=_candidate(), + proof_label="Proof A", + source_content="source", + source_title="title", + identification_agent=agent, + broadcast_fn=None, + ) + finally: + system_config.smt_enabled = previous + + assert hint is None + + +@pytest.mark.asyncio +@pytest.mark.parametrize("solver_result", ("unsat", "sat", "unknown", "invalid")) +async def test_only_unsat_suggests_lean_tactics(monkeypatch, solver_result) -> None: + stage = ProofVerificationStage() + + async def translate(**_kwargs): + return "(assert false)" + + async def check_smt2(_smtlib, timeout): + return SimpleNamespace( + result=solver_result, stdout=solver_result, stderr="", success=True + ) + + monkeypatch.setattr( + stage_module, "get_smt_client", lambda: SimpleNamespace(check_smt2=check_smt2) + ) + previous = system_config.smt_enabled + system_config.smt_enabled = True + try: + hint = await stage._run_smt_check( + user_prompt="Prove it", + source_type="brainstorm", + source_id="source", + base_event={}, + candidate=_candidate(), + proof_label="Proof A", + source_content="source", + source_title="title", + identification_agent=SimpleNamespace( + translate_candidate_to_smt=translate + ), + broadcast_fn=None, + ) + finally: + system_config.smt_enabled = previous + + assert hint is not None + assert bool(hint.suggested_tactics) is (solver_result == "unsat") + assert hint.result == (solver_result if solver_result != "invalid" else "unknown") + + +def test_smt_assistance_metadata_requires_successful_first_attempt_tactic_use() -> None: + hint = SmtHint(result="unsat", suggested_tactics=["omega"]) + rejected = ProofAttemptFeedback( + attempt=1, + theorem_id="t1", + lean_code="by omega", + success=False, + ) + accepted_without_hint = ProofAttemptFeedback( + attempt=1, + theorem_id="t1", + lean_code="by trivial", + success=True, + ) + accepted_with_hint = ProofAttemptFeedback( + attempt=1, + theorem_id="t1", + lean_code="by omega", + success=True, + ) + + assert not ProofVerificationStage._first_attempt_used_smt_hint([rejected], hint) + assert not ProofVerificationStage._first_attempt_used_smt_hint( + [accepted_without_hint], hint + ) + assert ProofVerificationStage._first_attempt_used_smt_hint( + [accepted_with_hint], hint + ) + assert not ProofVerificationStage._first_attempt_used_smt_hint( + [accepted_with_hint], + SmtHint(result="sat", suggested_tactics=["omega"]), + ) + + +@pytest.mark.asyncio +async def test_transient_smt_translation_failure_propagates_to_checkpoint_owner( + monkeypatch, +) -> None: + stage = ProofVerificationStage() + monkeypatch.setattr( + ProofVerificationStage, + "_is_smt_amenable", + staticmethod(lambda _candidate: True), + ) + monkeypatch.setattr( + stage_module.system_config, + "smt_enabled", + True, + ) + + async def fail_translation(**_kwargs): + raise RetryableProviderError( + provider="openrouter", + provider_label="OpenRouter", + role_id="proof", + model="test-model", + reason="temporary_disconnect", + message="temporary provider disconnect", + ) + + assert stage_module.system_config.smt_enabled is True + assert stage._is_smt_amenable(_candidate()) is True + with pytest.raises(RetryableProviderError): + await stage._run_smt_check( + user_prompt="Prove it", + source_type="brainstorm", + source_id="source", + base_event={}, + candidate=ProofCandidate( + theorem_id="t2", + statement="n : Nat satisfies n + 0 = n", + formal_sketch="Arithmetic equality", + ), + proof_label="Proof A", + source_content="source", + source_title="title", + identification_agent=SimpleNamespace( + translate_candidate_to_smt=fail_translation + ), + broadcast_fn=None, + ) diff --git a/tests/unit/test_solution_path_engine.py b/tests/unit/test_solution_path_engine.py new file mode 100644 index 0000000..0b5df72 --- /dev/null +++ b/tests/unit/test_solution_path_engine.py @@ -0,0 +1,604 @@ +import asyncio + +import pytest + +from backend.shared.solution_path import ( + ReviewDecision, + RouteEdit, + RouteStep, + SolutionPathEngine, + SolutionRoute, + StepOrdering, +) + + +def route(title: str, ordering: StepOrdering = StepOrdering.ORDERED) -> SolutionRoute: + return SolutionRoute(ordering=ordering, steps=[RouteStep(title=title)]) + + +@pytest.mark.asyncio +async def test_inactive_until_five_then_persists_one_plan(tmp_path): + calls = [] + + async def reviewer(proposal, plan): + calls.append((proposal.proposal_id, plan.revision if plan else 0)) + return ReviewDecision(decision="approve") + + engine = SolutionPathEngine(tmp_path, "run-a", reviewer) + await engine.propose(route("first")) + await engine.set_acceptance_count(4) + await asyncio.sleep(0) + assert calls == [] + assert engine.state.plan is None + + await engine.set_acceptance_count(5) + await engine.wait_idle() + assert engine.state.plan.route.steps[0].title == "first" + assert engine.state.plan.revision == 1 + + restored = SolutionPathEngine(tmp_path, "run-a", reviewer) + assert restored.state.plan.revision == 1 + assert restored.state.plan.route.ordering == StepOrdering.ORDERED + + +@pytest.mark.asyncio +async def test_acceptance_count_is_monotonic_and_activation_emits_once(tmp_path, monkeypatch): + events = [] + + async def reviewer(proposal, plan): + return ReviewDecision(decision="approve") + + engine = SolutionPathEngine(tmp_path, "run-monotonic", reviewer) + + async def capture(event, payload): + events.append(event) + + monkeypatch.setattr(engine, "_broadcast", capture) + await engine.set_acceptance_count(4) + await engine.set_acceptance_count(3) + assert engine.state.acceptance_count == 4 + + await asyncio.gather( + engine.set_acceptance_count(5), + engine.set_acceptance_count(7), + engine.set_acceptance_count(6), + ) + assert engine.state.acceptance_count == 7 + assert events.count("solution_path_activated") == 1 + + +@pytest.mark.asyncio +async def test_serial_followup_and_unordered_route(tmp_path): + active = 0 + max_active = 0 + seen = {} + + async def reviewer(proposal, plan): + nonlocal active, max_active + active += 1 + max_active = max(max_active, active) + await asyncio.sleep(0.01) + active -= 1 + count = seen.get(proposal.proposal_id, 0) + seen[proposal.proposal_id] = count + 1 + if count == 0 and proposal.route.steps[0].title == "follow": + return ReviewDecision( + decision="followup", + reasoning="revise", + followup_route=route("revised", StepOrdering.UNORDERED), + ) + return ReviewDecision(decision="approve") + + engine = SolutionPathEngine(tmp_path, "run-b", reviewer) + await engine.set_acceptance_count(5) + await engine.propose(route("follow")) + await engine.propose(route("second")) + await engine.wait_idle() + assert max_active == 1 + assert engine.state.plan.revision == 2 + assert engine.state.proposals[0].review_count == 2 + assert engine.state.proposals[0].route.ordering == StepOrdering.UNORDERED + assert [p.status.value for p in engine.state.proposals] == ["approved", "approved"] + + +@pytest.mark.asyncio +async def test_stale_revision_is_re_reviewed(tmp_path): + entered = asyncio.Event() + release = asyncio.Event() + calls = 0 + + async def reviewer(proposal, plan): + nonlocal calls + calls += 1 + if calls == 1: + entered.set() + await release.wait() + return ReviewDecision(decision="approve") + + engine = SolutionPathEngine(tmp_path, "run-c", reviewer) + await engine.set_acceptance_count(5) + await engine.propose(route("stale")) + await entered.wait() + # Simulate a concurrent durable plan revision while review is outside the lock. + engine._state.plan = engine._state.plan or __import__( + "backend.shared.solution_path", fromlist=["SolutionPlan"] + ).SolutionPlan(run_id="run-c", route=route("concurrent")) + release.set() + await engine.wait_idle() + assert calls == 2 + assert engine.state.plan.revision == 2 + assert any(a.event == "proposal_stale_requeued" for a in engine.state.audit) + + +@pytest.mark.asyncio +async def test_stop_preserves_and_clear_deletes(tmp_path): + entered = asyncio.Event() + + async def reviewer(proposal, plan): + entered.set() + await asyncio.Event().wait() + + engine = SolutionPathEngine(tmp_path, "run-d", reviewer) + await engine.set_acceptance_count(5) + await engine.propose(route("pending")) + await entered.wait() + await engine.stop() + state_path = tmp_path / "run-d" / "solution_path_state.json" + assert state_path.exists() + assert engine.state.proposals[0].status.value == "queued" + + await engine.clear() + assert not state_path.exists() + + +@pytest.mark.asyncio +async def test_scoped_lifecycle_queue_review_and_update_events(tmp_path, monkeypatch): + events = [] + + async def reviewer(proposal, plan): + return ReviewDecision(decision="approve") + + engine = SolutionPathEngine( + tmp_path, + "run-events", + reviewer, + workflow_mode="autonomous", + ) + + async def capture(event, payload): + events.append((event, payload)) + + monkeypatch.setattr(engine, "_broadcast", capture) + await engine.set_acceptance_count(5) + await engine.propose(route("event route")) + await engine.wait_idle() + + names = [event for event, _ in events] + assert names == [ + "solution_path_activated", + "solution_path_proposal_queued", + "solution_path_proposal_reviewing", + "solution_path_updated", + ] + for _, payload in events: + assert set(payload) == { + "workflow_mode", + "run_id", + "acceptance_count", + "prompt_hash", + "lifecycle_generation", + "revision", + "proposal_id", + "proposal_status", + "base_revision", + "decision", + "source_task_id", + "source_phase", + "source_decision", + "queued_proposals", + "reviewing_proposals", + "repair_required_proposals", + "repair_reason", + "message", + "detail", + } + assert payload["workflow_mode"] == "autonomous" + assert payload["run_id"] == "run-events" + assert payload["acceptance_count"] == 5 + assert payload["prompt_hash"] is None + assert isinstance(payload["revision"], int) + assert isinstance(payload["queued_proposals"], int) + assert isinstance(payload["reviewing_proposals"], int) + assert isinstance(payload["message"], str) + assert events[0][1]["proposal_id"] is None + assert events[1][1]["proposal_status"] == "queued" + assert events[2][1]["proposal_status"] == "reviewing" + assert events[3][1]["proposal_status"] == "approved" + + +@pytest.mark.asyncio +async def test_stop_start_resumes_same_run_and_pending_proposal(tmp_path): + first_entered = asyncio.Event() + resumed_calls = 0 + + async def blocked_reviewer(proposal, plan): + first_entered.set() + await asyncio.Event().wait() + + engine = SolutionPathEngine( + tmp_path, + "run-resume", + blocked_reviewer, + workflow_mode="autonomous", + user_prompt="Find a route", + ) + await engine.set_acceptance_count(5) + await engine.propose(route("pending")) + await first_entered.wait() + await engine.stop() + + async def resumed_reviewer(proposal, plan): + nonlocal resumed_calls + resumed_calls += 1 + return ReviewDecision(decision="approve") + + restored = SolutionPathEngine( + tmp_path, + "run-resume", + resumed_reviewer, + workflow_mode="autonomous", + user_prompt="Find a route", + ) + await restored.start() + await restored.wait_idle() + assert resumed_calls == 1 + assert restored.state.plan.route.steps[0].title == "pending" + assert restored.state.proposals[0].status.value == "approved" + + +@pytest.mark.asyncio +async def test_run_ownership_rejects_prompt_or_mode_mismatch(tmp_path): + async def reviewer(proposal, plan): + return ReviewDecision(decision="approve") + + owner = SolutionPathEngine( + tmp_path, + "owned-run", + reviewer, + workflow_mode="manual", + user_prompt="Original prompt", + ) + await owner.set_acceptance_count(1) + + with pytest.raises(ValueError, match="workflow_mode mismatch"): + SolutionPathEngine( + tmp_path, + "owned-run", + reviewer, + workflow_mode="leanoj", + user_prompt="Original prompt", + ) + with pytest.raises(ValueError, match="user_prompt mismatch"): + SolutionPathEngine( + tmp_path, + "owned-run", + reviewer, + workflow_mode="manual", + user_prompt="Different prompt", + ) + + +@pytest.mark.asyncio +async def test_retry_is_automatic_and_does_not_drop_proposal(tmp_path): + calls = 0 + + async def reviewer(proposal, plan): + nonlocal calls + calls += 1 + if calls == 1: + raise RuntimeError("temporary transport failure") + return ReviewDecision(decision="approve") + + engine = SolutionPathEngine( + tmp_path, + "run-retry", + reviewer, + retry_base_seconds=0.01, + retry_max_seconds=0.01, + ) + await engine.set_acceptance_count(5) + await engine.propose(route("retry me")) + await engine.wait_idle() + + assert calls == 2 + assert engine.state.plan.route.steps[0].title == "retry me" + assert engine.state.proposals[0].failure_count == 1 + assert any("retry_scheduled" in item.event for item in engine.state.audit) + + +@pytest.mark.asyncio +async def test_followup_supports_main_route_and_checked_partial_edits(tmp_path): + calls = 0 + + async def reviewer(proposal, plan): + nonlocal calls + calls += 1 + original = proposal.route.steps[0] + if calls == 1: + return ReviewDecision( + decision="followup", + reasoning="sharpen then review again", + main_route="Sharper overall route", + edits=[ + RouteEdit( + operation="check", + step_id=original.step_id, + expected_title=original.title, + ), + RouteEdit( + operation="update", + step_id=original.step_id, + step=__import__( + "backend.shared.solution_path", fromlist=["RouteStep"] + ).RouteStep(title="Sharpened step"), + ), + RouteEdit( + operation="add", + after_step_id=original.step_id, + step=__import__( + "backend.shared.solution_path", fromlist=["RouteStep"] + ).RouteStep(title="New step"), + ), + ], + more_edits=True, + ) + return ReviewDecision(decision="approve") + + engine = SolutionPathEngine(tmp_path, "run-edits", reviewer) + await engine.set_acceptance_count(5) + await engine.propose(route("Original"), main_route="Original route") + await engine.wait_idle() + + assert calls == 2 + assert engine.state.plan.main_route == "Sharper overall route" + assert [step.title for step in engine.state.plan.route.steps] == [ + "Sharpened step", + "New step", + ] + + +@pytest.mark.asyncio +async def test_audit_and_terminal_proposal_history_are_bounded(tmp_path): + async def reviewer(proposal, plan): + return ReviewDecision(decision="reject") + + engine = SolutionPathEngine(tmp_path, "run-bounds", reviewer) + await engine.set_acceptance_count(5) + for index in range(270): + await engine.propose(route(f"proposal-{index}")) + await engine.wait_idle() + + assert len(engine.state.proposals) <= 100 + assert len(engine.state.audit) <= 500 + sequences = [record.sequence for record in engine.state.audit] + assert sequences == sorted(sequences) + assert sequences[-1] > len(sequences) + + +@pytest.mark.asyncio +async def test_followups_continue_past_old_hidden_cap(tmp_path): + calls = 0 + + async def reviewer(proposal, plan): + nonlocal calls + calls += 1 + if calls <= 26: + return ReviewDecision( + decision="followup", + reasoning="continue material revision", + main_route=f"revision-{calls}", + ) + return ReviewDecision(decision="approve") + + engine = SolutionPathEngine(tmp_path, "run-long-followup", reviewer) + await engine.set_acceptance_count(5) + await engine.propose(route("Long revision")) + await engine.wait_idle() + + assert calls == 27 + assert engine.state.proposals[0].status.value == "approved" + assert engine.state.plan.main_route == "revision-26" + + +@pytest.mark.asyncio +async def test_context_overflow_becomes_terminal_user_repair(tmp_path): + async def reviewer(proposal, plan): + raise ValueError( + "solution-path reviewer mandatory context exceeds the configured input budget" + ) + + engine = SolutionPathEngine(tmp_path, "run-overflow", reviewer) + await engine.set_acceptance_count(5) + await engine.propose(route("Needs room")) + await engine.wait_idle() + + proposal = engine.state.proposals[0] + assert proposal.status.value == "user_repair_required" + assert proposal.next_retry_at is None + assert proposal.failure_count == 0 + assert proposal.repair_reason.value == "context_overflow" + + +@pytest.mark.asyncio +async def test_failed_later_edit_rolls_back_route_and_main_route(tmp_path): + calls = 0 + + async def reviewer(proposal, plan): + nonlocal calls + calls += 1 + if calls < 3: + original = proposal.route.steps[0] + return ReviewDecision( + decision="approve", + main_route="must not leak", + followup_route=SolutionRoute( + steps=[ + RouteStep( + step_id=original.step_id, + title="must not leak either", + ) + ] + ), + edits=[ + RouteEdit( + operation="update", + step_id=proposal.route.steps[0].step_id, + step=RouteStep(title="partial mutation"), + ), + RouteEdit( + operation="delete", + step_id="missing-later-step", + ), + ], + ) + return ReviewDecision(decision="reject", reasoning="stop") + + engine = SolutionPathEngine( + tmp_path, + "run-transaction", + reviewer, + retry_base_seconds=0.01, + retry_max_seconds=0.01, + ) + await engine.set_acceptance_count(5) + await engine.propose(route("original"), main_route="original route") + await engine.wait_idle() + + proposal = engine.state.proposals[0] + assert proposal.main_route == "original route" + assert [step.title for step in proposal.route.steps] == ["original"] + assert proposal.status.value == "rejected" + + +@pytest.mark.asyncio +async def test_hard_failure_requires_explicit_generation_fenced_resume( + tmp_path, monkeypatch +): + events = [] + calls = 0 + + async def reviewer(proposal, plan): + nonlocal calls + calls += 1 + if calls == 1: + raise RuntimeError("No OpenRouter API key is available") + return ReviewDecision(decision="approve") + + engine = SolutionPathEngine(tmp_path, "run-hard-repair", reviewer) + + async def capture(event, payload): + events.append((event, payload)) + + monkeypatch.setattr(engine, "_broadcast", capture) + await engine.set_acceptance_count(5) + await engine.propose(route("repair me")) + await engine.wait_idle() + + blocked = engine.state.proposals[0] + assert calls == 1 + assert blocked.status.value == "user_repair_required" + assert blocked.repair_reason.value == "missing_api_key" + + await engine.stop() + await engine.start() + await engine.wait_idle() + assert calls == 1 + + with pytest.raises(ValueError, match="stale"): + await engine.resume_proposal( + blocked.proposal_id, + lifecycle_generation=engine.state.lifecycle_generation - 1, + ) + await engine.resume_proposal( + blocked.proposal_id, + lifecycle_generation=engine.state.lifecycle_generation, + ) + await engine.wait_idle() + assert calls == 2 + assert engine.state.proposals[0].status.value == "approved" + assert "solution_path_proposal_resumed" in [event for event, _ in events] + + +@pytest.mark.asyncio +async def test_stale_detached_proposal_cannot_resurrect_cleared_state(tmp_path): + async def reviewer(proposal, plan): + return ReviewDecision(decision="approve") + + engine = SolutionPathEngine(tmp_path, "run-clear-fence", reviewer) + await engine.set_acceptance_count(5) + captured_generation = engine.state.lifecycle_generation + await engine.clear() + + with pytest.raises(ValueError, match="stale|stopped"): + await engine.propose( + route("stale"), + lifecycle_generation=captured_generation, + ) + assert not (tmp_path / "run-clear-fence" / "solution_path_state.json").exists() + + restored = SolutionPathEngine(tmp_path, "run-clear-fence", reviewer) + assert restored.state.lifecycle_generation > captured_generation + assert restored.state.proposals == [] + + +@pytest.mark.asyncio +async def test_persist_failure_rolls_back_in_memory_mutation(tmp_path, monkeypatch): + async def reviewer(proposal, plan): + return ReviewDecision(decision="approve") + + engine = SolutionPathEngine(tmp_path, "run-persist-rollback", reviewer) + original = engine.state + + def fail_write(payload): + raise OSError("disk unavailable") + + monkeypatch.setattr(engine, "_write_payload", fail_write) + with pytest.raises(OSError, match="disk unavailable"): + await engine.set_acceptance_count(5) + + assert engine.state == original + + +@pytest.mark.asyncio +async def test_unknown_failures_are_bounded_before_user_repair(tmp_path): + calls = 0 + + async def reviewer(proposal, plan): + nonlocal calls + calls += 1 + raise RuntimeError("opaque reviewer failure") + + engine = SolutionPathEngine( + tmp_path, + "run-bounded-failure", + reviewer, + retry_base_seconds=0.01, + retry_max_seconds=0.01, + ) + await engine.set_acceptance_count(5) + await engine.propose(route("bounded")) + await engine.wait_idle() + + proposal = engine.state.proposals[0] + assert calls == 3 + assert proposal.status.value == "user_repair_required" + assert proposal.repair_reason.value == "unknown_reviewer_failure" + + +def test_route_structure_and_metadata_are_bounded(): + from pydantic import ValidationError + + with pytest.raises(ValidationError): + SolutionRoute(steps=[RouteStep(title=str(index)) for index in range(33)]) + with pytest.raises(ValidationError): + RouteStep(title="x", metadata={"payload": "y" * 9000}) diff --git a/tests/unit/test_solution_path_integration.py b/tests/unit/test_solution_path_integration.py new file mode 100644 index 0000000..c6e890b --- /dev/null +++ b/tests/unit/test_solution_path_integration.py @@ -0,0 +1,227 @@ +from types import SimpleNamespace + +import pytest + +from backend.shared.solution_path.integration import ( + advisory_plan_block, + enqueue_optional_update, + validator_instruction, + with_budgeted_solver_plan, + with_validator_hook, +) +from backend.shared.solution_path.models import ( + SolutionPlan, + SolutionRoute, +) + + +class _Manager: + def __init__(self, *, active=True, plan=None): + self.active = active + self.state = SimpleNamespace(plan=plan) + self.proposals = [] + + async def propose( + self, + route, + *, + lifecycle_generation=0, + rationale="", + main_route="", + proposer_role="validator", + source_task_id=None, + source_phase=None, + source_decision=None, + ): + self.proposals.append( + ( + route, + rationale, + main_route, + proposer_role, + source_task_id, + source_phase, + source_decision, + ) + ) + return self.proposals[-1] + + +def test_plan_is_absent_before_activation(): + plan = SolutionPlan(run_id="run", route=SolutionRoute(steps=[])) + assert advisory_plan_block(_Manager(active=False, plan=plan)) == "" + prompt = "ordinary prompt" + assert with_validator_hook(prompt, _Manager(active=False, plan=plan)) == prompt + assert "solution_path_update" not in with_validator_hook( + prompt, _Manager(active=False, plan=plan) + ) + + +def test_budgeted_solver_plan_is_optional_when_context_is_full(): + plan = SolutionPlan( + run_id="run", + main_route="Use the direct route", + route=SolutionRoute(steps=[]), + ) + manager = _Manager(active=True, plan=plan) + prompt = "base prompt" + assert with_budgeted_solver_plan(prompt, manager, 1) == prompt + assert "[ADVISORY PROGRESSIVE SOLUTION PATH]" in with_budgeted_solver_plan( + prompt, manager, 1000 + ) + + +def test_validator_instruction_requests_one_optional_object(): + instruction = validator_instruction() + assert "`solution_path_update`" in instruction + assert "propose the initial path" in instruction + assert "absence of an existing path is not a reason" in instruction + assert "material revisions or meaningful progress" in instruction + assert "Omit the entire extension otherwise" in instruction + assert '"ordering":"ordered|parallel|mixed"' in instruction + assert "sole permitted extension" in instruction + batch_instruction = validator_instruction(batch=True) + assert "top level beside `decisions`" in batch_instruction + assert "never inside an individual decision" in batch_instruction + + +@pytest.mark.asyncio +async def test_optional_update_is_not_queued_before_activation(): + manager = _Manager(active=False) + assert await enqueue_optional_update( + { + "solution_path_update": { + "route": { + "ordering": "ordered", + "steps": [{"title": "Too early"}], + } + } + }, + manager, + ) is None + assert manager.proposals == [] + + +@pytest.mark.asyncio +async def test_malformed_optional_update_does_not_raise_or_enqueue(): + manager = _Manager() + assert await enqueue_optional_update( + {"decision": "accept", "solution_path_update": {"route": "bad"}}, + manager, + ) is None + assert manager.proposals == [] + + +@pytest.mark.asyncio +async def test_valid_optional_update_is_durably_enqueued(): + manager = _Manager() + proposal = await enqueue_optional_update( + { + "decision": "accept", + "solution_path_update": { + "main_route": "Resolve the central obstruction", + "route": { + "ordering": "ordered", + "steps": [{"title": "Resolve the obstruction"}], + }, + "rationale": "Materially sharper route", + }, + }, + manager, + proposer_role="agg_val", + source_task_id="agg_val_001", + source_phase="submission_validation", + source_decision="accept", + ) + assert proposal is not None + assert manager.proposals[0][1] == "Materially sharper route" + assert manager.proposals[0][2] == "Resolve the central obstruction" + assert manager.proposals[0][4:] == ( + "agg_val_001", + "submission_validation", + "accept", + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("ordering", ["parallel", "mixed"]) +async def test_parallel_optional_update_orderings_are_normalized(ordering): + manager = _Manager() + proposal = await enqueue_optional_update( + { + "solution_path_update": { + "main_route": "Pursue independent workstreams", + "route": { + "ordering": ordering, + "steps": [{"title": "Independent workstream"}], + }, + } + }, + manager, + ) + + assert proposal is not None + assert manager.proposals[0][0].ordering.value == "unordered" + + +@pytest.mark.asyncio +async def test_initial_update_requires_nonempty_main_route(): + manager = _Manager() + assert await enqueue_optional_update( + { + "solution_path_update": { + "route": { + "ordering": "ordered", + "steps": [{"title": "First route"}], + } + } + }, + manager, + ) is None + assert manager.proposals == [] + + +@pytest.mark.asyncio +async def test_later_update_omission_preserves_main_route(): + plan = SolutionPlan( + run_id="run", + main_route="Preserve this route", + route=SolutionRoute(steps=[]), + ) + manager = _Manager(plan=plan) + await enqueue_optional_update( + { + "solution_path_update": { + "route": { + "ordering": "ordered", + "steps": [{"title": "Refined step"}], + } + } + }, + manager, + ) + assert manager.proposals[0][2] == "Preserve this route" + + +def test_validator_and_solver_injection_use_only_active_canonical_plan(): + from backend.shared.solution_path.integration import ( + with_solver_plan, + with_validator_hook, + ) + + plan = SolutionPlan( + run_id="run", + main_route="Attack the obstruction", + route=SolutionRoute(steps=[]), + ) + inactive = _Manager(active=False, plan=plan) + active = _Manager(active=True, plan=plan) + + assert with_validator_hook("validator", inactive) == "validator" + assert with_solver_plan("solver", inactive) == "solver" + validator_prompt = with_validator_hook("validator", active) + solver_prompt = with_solver_plan("solver", active) + assert "Attack the obstruction" in validator_prompt + assert "solution_path_update" in validator_prompt + assert "Attack the obstruction" in solver_prompt + assert "solution_path_update" not in solver_prompt diff --git a/tests/unit/test_solution_path_registry.py b/tests/unit/test_solution_path_registry.py new file mode 100644 index 0000000..b205bfa --- /dev/null +++ b/tests/unit/test_solution_path_registry.py @@ -0,0 +1,251 @@ +import asyncio + +import pytest + +from backend.shared.solution_path import ( + ReviewDecision, + SolutionPathManagerRegistry, + stable_solution_path_run_id, +) + + +async def reviewer(proposal, plan): + return ReviewDecision(decision="approve") + + +@pytest.mark.asyncio +async def test_registry_reuses_paused_manager_and_preserves_provenance(tmp_path): + registry = SolutionPathManagerRegistry() + first = await registry.acquire( + tmp_path, + workflow_mode="manual", + user_prompt=" Prove the target ", + stable_run_id="manual", + reviewer=reviewer, + ) + await first.set_acceptance_count(7) + await first.stop() + + resumed = await registry.acquire( + tmp_path, + workflow_mode="manual", + user_prompt="Prove the target", + stable_run_id="manual", + reviewer=reviewer, + ) + assert resumed is first + assert resumed.state.acceptance_count == 7 + assert resumed.state.workflow_mode == "manual" + assert resumed.state.user_prompt == "Prove the target" + assert resumed.state.prompt_hash + + +@pytest.mark.asyncio +async def test_registry_rejects_prompt_collision_and_clear_deletes(tmp_path): + registry = SolutionPathManagerRegistry() + manager = await registry.acquire( + tmp_path, + workflow_mode="leanoj", + user_prompt="Prompt A", + stable_run_id="run-a", + reviewer=reviewer, + ) + state_path = tmp_path / "run-a" / "solution_path_state.json" + assert state_path.exists() + + with pytest.raises(ValueError, match="user_prompt mismatch"): + # A fresh registry exercises persisted provenance validation. + await SolutionPathManagerRegistry().acquire( + tmp_path, + workflow_mode="leanoj", + user_prompt="Prompt B", + stable_run_id="run-a", + reviewer=reviewer, + ) + + await registry.clear_manager(manager) + assert not state_path.exists() + assert registry.get("run-a") is None + + +@pytest.mark.asyncio +async def test_clear_run_preserves_unrelated_solution_path_histories(tmp_path): + registry = SolutionPathManagerRegistry() + manual = await registry.acquire( + tmp_path, + workflow_mode="manual", + user_prompt="Manual prompt", + stable_run_id="manual", + reviewer=reviewer, + ) + autonomous = await registry.acquire( + tmp_path, + workflow_mode="autonomous", + user_prompt="Autonomous prompt", + stable_run_id="session-a", + reviewer=reviewer, + ) + + await registry.clear_run(tmp_path, manual.run_id) + + assert not (tmp_path / "manual").exists() + assert (tmp_path / "session-a" / "solution_path_state.json").exists() + assert registry.get("session-a") is autonomous + + +def test_factory_identity_is_stable_and_mode_scoped(): + assert stable_solution_path_run_id("autonomous", " same ") == stable_solution_path_run_id( + "autonomous", "same" + ) + assert stable_solution_path_run_id("autonomous", "same") != stable_solution_path_run_id( + "leanoj", "same" + ) + + +@pytest.mark.asyncio +async def test_registry_canonicalizes_root_and_validates_loaded_provenance(tmp_path): + registry = SolutionPathManagerRegistry() + first = await registry.acquire( + tmp_path / "paths" / "..", + workflow_mode="manual", + user_prompt="Prompt A", + stable_run_id="same-run", + reviewer=reviewer, + ) + resumed = await registry.acquire( + tmp_path, + workflow_mode="manual", + user_prompt=" Prompt A ", + stable_run_id="same-run", + reviewer=reviewer, + ) + + assert resumed is first + with pytest.raises(ValueError, match="workflow_mode mismatch"): + await registry.acquire( + tmp_path, + workflow_mode="autonomous", + user_prompt="Prompt A", + stable_run_id="same-run", + reviewer=reviewer, + ) + with pytest.raises(ValueError, match="user_prompt mismatch"): + await registry.acquire( + tmp_path, + workflow_mode="manual", + user_prompt="Prompt B", + stable_run_id="same-run", + reviewer=reviewer, + ) + + +@pytest.mark.asyncio +async def test_clear_run_linearizes_with_reacquire_and_advances_generation( + tmp_path, monkeypatch +): + registry = SolutionPathManagerRegistry() + first = await registry.acquire( + tmp_path, + workflow_mode="manual", + user_prompt="Prompt", + stable_run_id="run", + reviewer=reviewer, + ) + clear_entered = asyncio.Event() + release_clear = asyncio.Event() + original_clear = first.clear + + async def blocked_clear(): + clear_entered.set() + await release_clear.wait() + await original_clear() + + monkeypatch.setattr(first, "clear", blocked_clear) + clear_task = asyncio.create_task(registry.clear_run(tmp_path, "run")) + await clear_entered.wait() + acquire_task = asyncio.create_task( + registry.acquire( + tmp_path, + workflow_mode="manual", + user_prompt="Prompt", + stable_run_id="run", + reviewer=reviewer, + ) + ) + await asyncio.sleep(0) + assert not acquire_task.done() + + lifecycle = await registry._lifecycle(tmp_path, "run") + generation_during_clear = lifecycle.generation + release_clear.set() + await clear_task + second = await acquire_task + + assert second is not first + assert lifecycle.generation == generation_during_clear + 1 + assert registry.get("run", tmp_path) is second + assert (tmp_path / "run" / "solution_path_state.json").exists() + + +@pytest.mark.asyncio +async def test_unrelated_run_lifecycle_does_not_share_lock(tmp_path, monkeypatch): + registry = SolutionPathManagerRegistry() + run_a = await registry.acquire( + tmp_path, + workflow_mode="manual", + user_prompt="A", + stable_run_id="run-a", + reviewer=reviewer, + ) + clear_entered = asyncio.Event() + release_clear = asyncio.Event() + original_clear = run_a.clear + + async def blocked_clear(): + clear_entered.set() + await release_clear.wait() + await original_clear() + + monkeypatch.setattr(run_a, "clear", blocked_clear) + clear_task = asyncio.create_task(registry.clear_run(tmp_path, "run-a")) + await clear_entered.wait() + + run_b = await asyncio.wait_for( + registry.acquire( + tmp_path, + workflow_mode="autonomous", + user_prompt="B", + stable_run_id="run-b", + reviewer=reviewer, + ), + timeout=1, + ) + assert registry.get("run-b", tmp_path) is run_b + + release_clear.set() + await clear_task + + +@pytest.mark.asyncio +async def test_same_run_id_is_independent_across_roots(tmp_path): + registry = SolutionPathManagerRegistry() + first = await registry.acquire( + tmp_path / "one", + workflow_mode="manual", + user_prompt="One", + stable_run_id="shared", + reviewer=reviewer, + ) + second = await registry.acquire( + tmp_path / "two", + workflow_mode="autonomous", + user_prompt="Two", + stable_run_id="shared", + reviewer=reviewer, + ) + + assert first is not second + assert registry.get("shared", tmp_path / "one") is first + assert registry.get("shared", tmp_path / "two") is second + with pytest.raises(ValueError, match="ambiguous"): + registry.get("shared") diff --git a/tests/unit/test_solution_path_reviewer.py b/tests/unit/test_solution_path_reviewer.py new file mode 100644 index 0000000..0ff66c6 --- /dev/null +++ b/tests/unit/test_solution_path_reviewer.py @@ -0,0 +1,111 @@ +import json + +import pytest + +from backend.shared.solution_path import ( + PlanProposal, + RouteStep, + SolutionRoute, + build_review_prompt, + compact_review_prompt, + review_with_json_retry, +) + + +def _proposal() -> PlanProposal: + return PlanProposal( + run_id="run", + base_revision=0, + route=SolutionRoute(steps=[RouteStep(title="First")]), + ) + + +@pytest.mark.asyncio +async def test_reviewer_repairs_malformed_json_with_sanitized_retry(): + prompts = [] + responses = [ + {"text": "<|channel>analysis broken output"}, + { + "text": json.dumps( + { + "decision": "followup", + "reasoning": "add one step", + "edits": [ + { + "operation": "add", + "step": {"title": "Second"}, + } + ], + "more_edits": True, + } + ) + }, + ] + + async def call(messages): + prompts.append(messages) + return responses.pop(0) + + decision = await review_with_json_retry( + prompt=build_review_prompt( + user_prompt="Solve it", + proposal=_proposal(), + current_plan=None, + ), + call_completion=call, + extract_text=lambda response: response["text"], + context_window=10000, + max_output_tokens=1000, + ) + + assert decision.decision == "followup" + assert decision.edits[0].operation == "add" + assert len(prompts) == 2 + assert "<|channel>analysis" not in str(prompts[1]) + + +@pytest.mark.asyncio +async def test_reviewer_rejects_mandatory_context_overflow_before_call(): + called = False + + async def call(messages): + nonlocal called + called = True + return {"text": "{}"} + + with pytest.raises(ValueError, match="mandatory context exceeds"): + await review_with_json_retry( + prompt="large " * 1000, + call_completion=call, + extract_text=lambda response: response["text"], + context_window=100, + max_output_tokens=50, + ) + assert called is False + + +@pytest.mark.asyncio +async def test_reviewer_uses_compacted_prompt_before_terminal_overflow(): + seen = [] + + async def call(messages): + seen.append(messages[0]["content"]) + return {"text": '{"decision":"approve","reasoning":"fits"}'} + + proposal = _proposal() + compact = compact_review_prompt( + user_prompt="Solve it", + proposal=proposal, + current_plan=None, + ) + decision = await review_with_json_retry( + prompt=("bookkeeping " * 5000) + compact, + compact_prompt=compact, + call_completion=call, + extract_text=lambda response: response["text"], + context_window=2000, + max_output_tokens=500, + ) + + assert decision.decision == "approve" + assert seen == [compact] diff --git a/tests/unit/test_solution_path_role_classification.py b/tests/unit/test_solution_path_role_classification.py new file mode 100644 index 0000000..e5591f9 --- /dev/null +++ b/tests/unit/test_solution_path_role_classification.py @@ -0,0 +1,82 @@ +from types import SimpleNamespace +from unittest.mock import AsyncMock + +import pytest + +from backend.leanoj.core.leanoj_coordinator import ( + LeanOJCoordinator, + _solution_path_call_kind, +) +from backend.shared.models import LeanOJRoleConfig, LeanOJStartRequest + + +def test_leanoj_semantic_validator_classification_is_explicit(): + assert _solution_path_call_kind("leanoj_topic_val", "leanoj_topic_validator") == "validator" + assert _solution_path_call_kind("leanoj_final_review", "leanoj_final_solver") == "validator" + assert ( + _solution_path_call_kind( + "leanoj_master_proof_edit_val", "leanoj_master_proof_edit_validator" + ) + == "validator" + ) + + +def test_leanoj_solver_and_excluded_classification(): + assert _solution_path_call_kind("leanoj_final", "leanoj_final_solver") == "solver" + assert ( + _solution_path_call_kind( + "leanoj_brainstorm", "leanoj_brainstorm_submitter_10" + ) + == "solver" + ) + assert _solution_path_call_kind("proof_lean", "leanoj_proof_novelty") == "excluded" + assert _solution_path_call_kind("tool_call", "leanoj_brainstorm_validator") == "excluded" + + +@pytest.mark.asyncio +async def test_leanoj_solution_path_uses_dedicated_main_submitter_role( + monkeypatch, tmp_path +): + import backend.shared.solution_path as solution_path + from backend.leanoj.core import leanoj_coordinator as module + + primary = LeanOJRoleConfig( + model_id="main-model", + context_window=4096, + max_output_tokens=512, + ) + secondary = LeanOJRoleConfig( + model_id="secondary-model", + context_window=8192, + max_output_tokens=1024, + ) + request = LeanOJStartRequest( + user_prompt="Solve the template.", + lean_template="theorem target : True := by sorry", + topic_generator=primary, + topic_validator=primary, + brainstorm_submitters=[primary, secondary], + brainstorm_validator=primary, + final_solver=primary, + ) + coordinator = LeanOJCoordinator() + coordinator._state.session_id = "run" + configured = {} + monkeypatch.setattr( + coordinator, + "_configure_role", + lambda role_id, config: configured.setdefault(role_id, config), + ) + monkeypatch.setattr(module.system_config, "data_dir", tmp_path) + manager = SimpleNamespace( + set_acceptance_count=AsyncMock(), + ) + + async def acquire(*_args, **_kwargs): + assert configured["leanoj_solution_path_reviewer"] is primary + return manager + + monkeypatch.setattr(solution_path.solution_path_registry, "acquire", acquire) + await coordinator._initialize_solution_path_manager(request) + + assert configured["leanoj_solution_path_reviewer"].model_id == "main-model" diff --git a/tests/unit/test_topic_selection_models.py b/tests/unit/test_topic_selection_models.py new file mode 100644 index 0000000..b5790b5 --- /dev/null +++ b/tests/unit/test_topic_selection_models.py @@ -0,0 +1,30 @@ +from backend.shared.models import FreeModelSettings, TopicSelectionSubmission + + +def test_topic_selection_accepts_new_topic_action() -> None: + submission = TopicSelectionSubmission( + action="new_topic", + topic_prompt="Attack the next direct route to the user's goal", + reasoning="This opens a distinct useful avenue.", + ) + + assert submission.action == "new_topic" + assert submission.topic_prompt + + +def test_topic_selection_accepts_continue_existing_action() -> None: + submission = TopicSelectionSubmission( + action="continue_existing", + topic_id="topic_003", + reasoning="The existing brainstorm is incomplete and still valuable.", + ) + + assert submission.action == "continue_existing" + assert submission.topic_id == "topic_003" + + +def test_free_model_settings_default_fallback_controls_disabled() -> None: + settings = FreeModelSettings() + + assert settings.looping_enabled is False + assert settings.auto_selector_enabled is False diff --git a/tests/test_wolfram_tool_loop.py b/tests/unit/test_wolfram_tool_loop.py similarity index 73% rename from tests/test_wolfram_tool_loop.py rename to tests/unit/test_wolfram_tool_loop.py index ef35c66..b166949 100644 --- a/tests/test_wolfram_tool_loop.py +++ b/tests/unit/test_wolfram_tool_loop.py @@ -2,7 +2,10 @@ import unittest from backend.compiler.agents import writer_submitter as submitter_module -from backend.compiler.agents.writer_submitter import WritingSubmitter +from backend.compiler.agents.writer_submitter import ( + WolframFinalizationError, + WritingSubmitter, +) from backend.shared import wolfram_alpha_client as wolfram_module from backend.shared.config import system_config @@ -128,6 +131,49 @@ async def fake_generate_completion(**kwargs): self.assertEqual(fake_client.queries, ["1+1", "2+2"]) self.assertEqual(len(wolfram_calls), 2) + async def test_repeated_tool_calls_after_forced_finalization_fail_specifically( + self, + ) -> None: + submitter = WritingSubmitter(model_name="test-model", user_prompt="Write.") + fake_client = FakeWolframClient() + + async def fake_generate_completion(**_kwargs): + return { + "choices": [ + { + "message": { + "content": "stale assistant preamble", + "tool_calls": [_tool_call("again", "2+2")], + } + } + ] + } + + original_available = submitter_module._wolfram_tool_available + original_get_client = wolfram_module.get_wolfram_client + original_generate = submitter_module.api_client_manager.generate_completion + original_budget = submitter_module.WOLFRAM_MAX_CALLS_PER_SUBMISSION + try: + submitter_module._wolfram_tool_available = lambda: True + wolfram_module.get_wolfram_client = lambda: fake_client + submitter_module.api_client_manager.generate_completion = ( + fake_generate_completion + ) + submitter_module.WOLFRAM_MAX_CALLS_PER_SUBMISSION = 1 + + with self.assertRaisesRegex( + WolframFinalizationError, "did not produce final JSON" + ): + await submitter._generate_completion_with_wolfram_tool( + task_id="task-cap", + initial_prompt="prompt", + ) + finally: + submitter_module._wolfram_tool_available = original_available + wolfram_module.get_wolfram_client = original_get_client + submitter_module.api_client_manager.generate_completion = original_generate + submitter_module.WOLFRAM_MAX_CALLS_PER_SUBMISSION = original_budget + if __name__ == "__main__": unittest.main() diff --git a/tests/workflow_browser_smoke/__init__.py b/tests/workflow_browser_smoke/__init__.py new file mode 100644 index 0000000..95c4d61 --- /dev/null +++ b/tests/workflow_browser_smoke/__init__.py @@ -0,0 +1 @@ +"""Browser smoke coverage metadata and Playwright specifications.""" diff --git a/tests/workflow_browser_smoke/autonomous-controls.spec.js b/tests/workflow_browser_smoke/autonomous-controls.spec.js new file mode 100644 index 0000000..336d215 --- /dev/null +++ b/tests/workflow_browser_smoke/autonomous-controls.spec.js @@ -0,0 +1,33 @@ +import { expect, test } from './fixtures.js'; + +test('autonomous start sends hosted payload, locks controls, and stops cleanly', async ({ page, mockApp }) => { + await mockApp.open(); + + const prompt = page.getByLabel('Research Goal'); + await prompt.fill('Prove a deterministic browser smoke theorem.'); + await page.getByRole('button', { name: 'Start Research' }).click(); + + await expect.poll(() => mockApp.requests('POST', '/api/auto-research/start').length).toBe(1); + const startPayload = mockApp.requests('POST', '/api/auto-research/start')[0].body; + expect(startPayload.user_research_prompt).toBe('Prove a deterministic browser smoke theorem.'); + expect(startPayload.allow_mathematical_proofs).toBe(false); + expect(startPayload.allow_research_papers).toBe(true); + expect(startPayload.submitter_configs.length).toBeGreaterThan(0); + expect(startPayload.submitter_configs.every((role) => role.provider === 'openrouter')).toBe(true); + expect(startPayload.submitter_configs.every((role) => role.lm_studio_fallback_id === null)).toBe(true); + expect(startPayload.validator_provider).toBe('openrouter'); + expect(startPayload.writer_provider).toBe('openrouter'); + expect(startPayload.high_param_provider).toBe('openrouter'); + + await expect(prompt).toBeDisabled(); + await expect(page.getByText('Running', { exact: true })).toBeVisible(); + const stopButton = page.getByRole('button', { name: 'Stop Research' }); + await expect(stopButton).toBeVisible(); + await stopButton.click(); + + await expect.poll(() => mockApp.requests('POST', '/api/auto-research/stop').length).toBe(1); + await expect(page.getByRole('button', { name: 'Start Research' })).toBeVisible(); + await expect(prompt).toBeEnabled(); + + await mockApp.assertNoUnexpectedRequests(); +}); diff --git a/tests/workflow_browser_smoke/coverage_records.py b/tests/workflow_browser_smoke/coverage_records.py new file mode 100644 index 0000000..03cb40d --- /dev/null +++ b/tests/workflow_browser_smoke/coverage_records.py @@ -0,0 +1,97 @@ +from __future__ import annotations + +from tests.workflow_harness.coverage_metadata import InteractionCoverage + + +BROWSER_SMOKE_COVERAGE = ( + InteractionCoverage( + scenario_id="browser_hosted_startup_respects_desktop_capability_boundaries", + fields=("websocket_api_contracts", "proof_runtime_gating", "runtime_exclusivity"), + invariants=("api.hosted_desktop_only_routes_unavailable",), + adapter="browser_smoke", + result="passed", + test_file="tests/workflow_browser_smoke/hosted-capabilities.spec.js", + evidence=( + "hosted_openrouter_startup", + "desktop_provider_paths_not_requested", + ), + runner="playwright", + asserted_invariants=("api.hosted_desktop_only_routes_unavailable",), + test_selectors=( + "tests/workflow_browser_smoke/hosted-capabilities.spec.js::hosted capabilities drive startup copy and hide desktop-only paths", + ), + ), + InteractionCoverage( + scenario_id="browser_autonomous_start_stop_preserves_single_ui_owner", + fields=("runtime_exclusivity", "allowed_outputs", "websocket_api_contracts"), + invariants=( + "runtime.single_active_workflow", + "outputs.at_least_one_output_enabled", + ), + adapter="browser_smoke", + result="passed", + test_file="tests/workflow_browser_smoke/autonomous-controls.spec.js", + evidence=( + "papers_only_start_payload", + "running_owner_controls", + "stop_restores_controls", + ), + runner="playwright", + asserted_invariants=( + "runtime.single_active_workflow", + "outputs.at_least_one_output_enabled", + ), + test_selectors=( + "tests/workflow_browser_smoke/autonomous-controls.spec.js::autonomous start sends hosted payload, locks controls, and stops cleanly", + ), + ), + InteractionCoverage( + scenario_id="browser_mode_switching_preserves_in_memory_default_and_developer_gate", + fields=("runtime_exclusivity", "workflow_filesystem_state", "websocket_api_contracts"), + invariants=("runtime.single_active_workflow",), + adapter="browser_smoke", + result="passed", + test_file="tests/workflow_browser_smoke/mode-switching.spec.js", + evidence=( + "autonomous_default_mode", + "manual_mode_switch", + "leanoj_hidden_before_developer_gate", + "leanoj_visible_after_developer_gate", + "reload_restores_autonomous_mode", + ), + runner="playwright", + asserted_invariants=("runtime.single_active_workflow",), + test_selectors=( + "tests/workflow_browser_smoke/mode-switching.spec.js::mode switching is in-memory and LeanOJ remains developer gated", + ), + ), + InteractionCoverage( + scenario_id="browser_overflow_activity_persists_with_attribution_and_terminal_deduplication", + fields=("websocket_api_contracts", "workflow_filesystem_state", "provider_pause_resume"), + invariants=( + "events.context_overflow_route_identity", + "events.context_overflow_persists_across_reload", + "events.context_overflow_terminal_stop_once", + "events.proof_context_overflow_nonfatal", + ), + adapter="browser_smoke", + result="passed", + test_file="tests/workflow_browser_smoke/overflow-activity.spec.js", + evidence=( + "configured_and_effective_route_visible", + "proof_overflow_keeps_parent_active", + "fatal_stop_not_duplicated", + "activity_restored_after_reload", + ), + runner="playwright", + asserted_invariants=( + "events.context_overflow_route_identity", + "events.context_overflow_persists_across_reload", + "events.context_overflow_terminal_stop_once", + "events.proof_context_overflow_nonfatal", + ), + test_selectors=( + "tests/workflow_browser_smoke/overflow-activity.spec.js::overflow activity attributes routes, keeps proof overflow nonfatal, dedupes stop, and persists", + ), + ), +) diff --git a/tests/workflow_browser_smoke/fixtures.js b/tests/workflow_browser_smoke/fixtures.js new file mode 100644 index 0000000..689c468 --- /dev/null +++ b/tests/workflow_browser_smoke/fixtures.js @@ -0,0 +1,193 @@ +// Browser specs live with the cross-field test artifacts while their locked Node +// runner is owned by the frontend package. +import { expect, test as base } from '../../frontend/node_modules/@playwright/test/index.js'; + +const HOSTED_FEATURES = { + version: 'browser-smoke', + build_commit: 'browser-smoke-commit', + update_channel: 'main', + api_contract_version: 'build-g-smoke', + generic_mode: true, + lm_studio_enabled: false, + pdf_download_available: false, + openai_codex_oauth_available: false, + xai_grok_oauth_available: false, + sakana_fugu_available: false, +}; + +const IDLE_STATUS = { + is_running: false, + current_tier: null, + is_tier3_active: false, + tier3_status: null, +}; + +const json = (route, body, status = 200) => route.fulfill({ + status, + contentType: 'application/json', + body: JSON.stringify(body), +}); + +function responseFor(method, path, state) { + if (method === 'GET' && path === '/api/features') return state.features; + if (method === 'GET' && path === '/api/openrouter/api-key-status') { + return { success: true, has_key: state.hasOpenRouterKey, enabled: state.hasOpenRouterKey }; + } + if (method === 'GET' && path === '/api/cloud-access/status') return { providers: {} }; + if (method === 'GET' && path === '/api/cloud-access/provider-notifications') return { notifications: [] }; + if (method === 'GET' && path === '/api/connectivity/status') { + return { + inference: { + openrouter_oauth: { status: state.hasOpenRouterKey ? 'ready' : 'inactive' }, + lm_studio: { status: 'inactive' }, + }, + skills: { + syntheticlib4: { status: 'Coming soon', enabled: false }, + agent_conversation_memory: { status: 'ready', enabled: true }, + wolfram_alpha: { status: 'inactive', enabled: false }, + }, + }; + } + if (method === 'GET' && path === '/api/update-notice') return { update_available: false }; + if (method === 'GET' && path === '/api/aggregator/models') return { models: [] }; + if (method === 'GET' && path === '/api/aggregator/status') return { is_running: false }; + if (method === 'GET' && path === '/api/aggregator/prompt') return { prompt: '' }; + if (method === 'GET' && path === '/api/compiler/status') return { is_running: false }; + if (method === 'GET' && path === '/api/leanoj/status') return { is_running: false, phase: 'idle' }; + if (method === 'GET' && path === '/api/auto-research/status') return state.autonomousStatus; + if (method === 'GET' && path === '/api/auto-research/prompt') return { prompt: '' }; + if (method === 'GET' && path === '/api/auto-research/brainstorms') return { brainstorms: [] }; + if (method === 'GET' && path === '/api/auto-research/papers') return { papers: [] }; + if (method === 'GET' && path === '/api/auto-research/stats') { + return { total_papers_completed: 0, paper_counts: { active: 0, pruned: 0 } }; + } + if (method === 'GET' && path === '/api/auto-research/current-session') return { session: null }; + if (method === 'GET' && path === '/api/auto-research/tier3/status') return { is_active: false, status: 'idle' }; + if (method === 'GET' && path === '/api/auto-research/tier3/final-answer') return { has_final_answer: false }; + if (method === 'GET' && path === '/api/auto-research/tier3/volume-progress') return { is_long_form: false }; + if (method === 'GET' && path === '/api/proofs/status') { + return { lean4_enabled: false, smt_enabled: false, manual_check_ready: false }; + } + if (method === 'GET' && path === '/api/proof-search/assistant/latest-pack') { + return { available: false, supports: [] }; + } + if (method === 'GET' && path === '/api/workflow/predictions') return { mode: null, tasks: [] }; + if (method === 'GET' && path === '/api/workflow/solution-path') return { active: false, state: null }; + if (method === 'GET' && path === '/api/token-stats') { + return { total_input: 0, total_output: 0, by_model: {}, elapsed_seconds: 0 }; + } + if (method === 'GET' && path === '/api/boost/status') { + return { enabled: false, boost_next_count: 0, always_prefer: false, boosted_categories: [] }; + } + if (method === 'GET' && path === '/api/boost/categories') return { categories: [] }; + if (method === 'POST' && path === '/api/auto-research/start') { + state.autonomousStatus = { ...IDLE_STATUS, is_running: true, current_tier: 'topic_exploration' }; + return { success: true, status: 'started' }; + } + if (method === 'POST' && path === '/api/auto-research/stop') { + state.autonomousStatus = { ...IDLE_STATUS }; + return { success: true, status: 'stopped' }; + } + return undefined; +} + +export const test = base.extend({ + mockApp: async ({ page }, use) => { + const state = { + features: { ...HOSTED_FEATURES }, + hasOpenRouterKey: true, + autonomousStatus: { ...IDLE_STATUS }, + requests: [], + unexpectedRequests: [], + }; + + await page.addInitScript(() => { + class MockWebSocket { + static OPEN = 1; + static CLOSED = 3; + static instances = []; + + constructor(url) { + this.url = url; + this.readyState = MockWebSocket.OPEN; + MockWebSocket.instances.push(this); + queueMicrotask(() => this.onopen?.({ type: 'open' })); + } + + send() {} + + close() { + this.readyState = MockWebSocket.CLOSED; + this.onclose?.({ type: 'close' }); + } + + serverSend(message) { + this.onmessage?.({ data: JSON.stringify(message) }); + } + } + + window.WebSocket = MockWebSocket; + window.__motoBrowserSmoke = { + send(type, data, timestamp = '2026-07-14T12:00:00.000Z') { + const socket = MockWebSocket.instances.at(-1); + if (!socket) throw new Error('MOTO WebSocket has not connected'); + socket.serverSend({ type, data, timestamp }); + }, + }; + }); + + await page.route('**/*', async (route) => { + const request = route.request(); + const url = new URL(request.url()); + const isLocal = ['127.0.0.1', 'localhost'].includes(url.hostname); + if (!isLocal) { + state.unexpectedRequests.push(`${request.method()} ${request.url()}`); + await route.abort('blockedbyclient'); + return; + } + if (!url.pathname.startsWith('/api/')) { + await route.continue(); + return; + } + + const entry = { + method: request.method(), + path: `${url.pathname}${url.search}`, + body: request.postDataJSON?.() ?? null, + }; + state.requests.push(entry); + const response = responseFor(request.method(), url.pathname, state); + if (response === undefined) { + state.unexpectedRequests.push(`${request.method()} ${url.pathname}`); + await json(route, { detail: `Unmocked browser-smoke API: ${request.method()} ${url.pathname}` }, 501); + return; + } + await json(route, response); + }); + + const api = { + state, + async open({ acknowledgeDisclaimer = true } = {}) { + await page.goto('/'); + if (acknowledgeDisclaimer) { + await page.getByRole('button', { name: 'I Have Read and Acknowledge This Disclaimer' }).click(); + } + }, + requests(method, path) { + return state.requests.filter((request) => request.method === method && request.path === path); + }, + async sendWebSocket(type, data, timestamp) { + await page.evaluate(({ eventType, payload, eventTimestamp }) => { + window.__motoBrowserSmoke.send(eventType, payload, eventTimestamp); + }, { eventType: type, payload: data, eventTimestamp: timestamp }); + }, + async assertNoUnexpectedRequests() { + expect(state.unexpectedRequests).toEqual([]); + }, + }; + + await use(api); + }, +}); + +export { expect }; diff --git a/tests/workflow_browser_smoke/hosted-capabilities.spec.js b/tests/workflow_browser_smoke/hosted-capabilities.spec.js new file mode 100644 index 0000000..6e33e14 --- /dev/null +++ b/tests/workflow_browser_smoke/hosted-capabilities.spec.js @@ -0,0 +1,19 @@ +import { expect, test } from './fixtures.js'; + +test('hosted capabilities drive startup copy and hide desktop-only paths', async ({ page, mockApp }) => { + mockApp.state.hasOpenRouterKey = false; + await mockApp.open({ acknowledgeDisclaimer: false }); + + await expect(page.getByText('This hosted deployment uses OpenRouter-only inference.')).toBeVisible(); + await expect(page.getByText('LM Studio is intentionally disabled in this environment.')).toBeVisible(); + + await page.getByRole('button', { name: 'I Have Read and Acknowledge This Disclaimer' }).click(); + + await expect(page.getByRole('heading', { name: 'Choose Your Startup Setup' })).toBeVisible(); + await expect(page.getByText('This hosted deployment needs an OpenRouter API key before you start.')).toBeVisible(); + await expect(page.getByRole('button', { name: 'Enter OpenRouter Key' })).toBeVisible(); + await expect(page.getByRole('heading', { name: 'LM Studio Setup' })).toHaveCount(0); + await expect(page.getByText('Cloud Provider Add-On')).toHaveCount(0); + + await mockApp.assertNoUnexpectedRequests(); +}); diff --git a/tests/workflow_browser_smoke/mode-switching.spec.js b/tests/workflow_browser_smoke/mode-switching.spec.js new file mode 100644 index 0000000..ff2aff5 --- /dev/null +++ b/tests/workflow_browser_smoke/mode-switching.spec.js @@ -0,0 +1,31 @@ +import { expect, test } from './fixtures.js'; + +test('mode switching is in-memory and LeanOJ remains developer gated', async ({ page, mockApp }) => { + await mockApp.open(); + + const modeSelect = page.locator('#app-mode-select'); + await expect(modeSelect).toHaveValue('autonomous'); + await expect(modeSelect.getByRole('option', { name: 'LeanOJ Proof Solver' })).toHaveCount(0); + + await modeSelect.selectOption('manual'); + await expect(page.getByText('Manual S.T.E.M. Writer')).toBeVisible(); + + await page.keyboard.down('Shift'); + await page.keyboard.down('z'); + await page.keyboard.down('x'); + await page.keyboard.up('x'); + await page.keyboard.up('z'); + await page.keyboard.up('Shift'); + + await expect(modeSelect.getByRole('option', { name: 'LeanOJ Proof Solver' })).toHaveCount(1); + await modeSelect.selectOption('leanoj'); + await expect(page.getByText('Proof Solver Mode', { exact: true })).toBeVisible(); + + await page.reload(); + await expect(page.getByRole('button', { name: 'I Have Read and Acknowledge This Disclaimer' })).toBeVisible(); + await page.getByRole('button', { name: 'I Have Read and Acknowledge This Disclaimer' }).click(); + await expect(page.locator('#app-mode-select')).toHaveValue('autonomous'); + await expect(page.getByRole('heading', { name: 'Autonomous Research' })).toBeVisible(); + + await mockApp.assertNoUnexpectedRequests(); +}); diff --git a/tests/workflow_browser_smoke/overflow-activity.spec.js b/tests/workflow_browser_smoke/overflow-activity.spec.js new file mode 100644 index 0000000..ca8dcf7 --- /dev/null +++ b/tests/workflow_browser_smoke/overflow-activity.spec.js @@ -0,0 +1,47 @@ +import { expect, test } from './fixtures.js'; + +test('overflow activity attributes routes, keeps proof overflow nonfatal, dedupes stop, and persists', async ({ page, mockApp }) => { + await mockApp.open(); + + const proofMessage = 'Proof candidate exceeded its direct-context budget'; + await mockApp.sendWebSocket('proof_context_overflow', { + workflow_mode: 'autonomous', + role_id: 'autonomous_proof_formalization', + message: proofMessage, + configured_model: 'configured-proof-model', + configured_provider: 'openrouter', + effective_model: 'rotated-proof-model', + effective_provider: 'openrouter', + effective_host_provider: 'Provider A', + }); + + await expect(page.getByText(new RegExp(`${proofMessage}.*Effective route: rotated-proof-model via openrouter, host Provider A.*Configured route: configured-proof-model via openrouter`))).toBeVisible(); + await expect(page.getByRole('button', { name: 'Start Research' })).toBeVisible(); + + const terminalMessage = 'Autonomous direct context exceeded the selected model window'; + await mockApp.sendWebSocket('context_overflow_error', { + workflow_mode: 'autonomous', + role_id: 'agg_sub1_topic', + message: terminalMessage, + configured_model: 'configured-topic-model', + configured_provider: 'openrouter', + effective_model: 'fallback-topic-model', + effective_provider: 'lm_studio', + }, '2026-07-14T12:01:00.000Z'); + await mockApp.sendWebSocket('auto_research_stopped', { + reason: 'context_overflow', + message: terminalMessage, + }, '2026-07-14T12:01:01.000Z'); + + const terminalActivity = page.locator('.activity-message').filter({ hasText: terminalMessage }); + await expect(terminalActivity).toHaveCount(1); + await expect(terminalActivity).toContainText('Effective route: fallback-topic-model via lm_studio'); + await expect(terminalActivity).toContainText('Configured route: configured-topic-model via openrouter'); + + await page.reload(); + await page.getByRole('button', { name: 'I Have Read and Acknowledge This Disclaimer' }).click(); + await expect(page.locator('.activity-message').filter({ hasText: proofMessage })).toHaveCount(1); + await expect(page.locator('.activity-message').filter({ hasText: terminalMessage })).toHaveCount(1); + + await mockApp.assertNoUnexpectedRequests(); +}); diff --git a/tests/workflow_browser_smoke/test_coverage_metadata.py b/tests/workflow_browser_smoke/test_coverage_metadata.py new file mode 100644 index 0000000..e3dc674 --- /dev/null +++ b/tests/workflow_browser_smoke/test_coverage_metadata.py @@ -0,0 +1,30 @@ +from pathlib import Path + +from tests.workflow_browser_smoke.coverage_records import BROWSER_SMOKE_COVERAGE +from tests.workflow_harness.coverage_metadata import assert_coverage_records_valid + + +def test_browser_smoke_coverage_metadata_is_valid(): + assert_coverage_records_valid(BROWSER_SMOKE_COVERAGE) + + +def test_passed_browser_coverage_links_exact_playwright_tests_and_invariants(): + for record in BROWSER_SMOKE_COVERAGE: + assert record.runner == "playwright" + assert len(record.test_selectors) == 1 + assert record.test_selectors[0].startswith(f"{record.test_file}::") + assert set(record.asserted_invariants) == set(record.invariants) + + +def test_every_browser_smoke_spec_has_exactly_one_coverage_record(): + smoke_dir = Path(__file__).resolve().parent + repository_root = smoke_dir.parents[1] + expected_test_files = { + path.relative_to(repository_root).as_posix() + for path in smoke_dir.glob("*.spec.js") + } + represented_test_files = [record.test_file for record in BROWSER_SMOKE_COVERAGE] + + assert len(represented_test_files) == len(set(represented_test_files)) + assert set(represented_test_files) == expected_test_files + assert all((repository_root / test_file).is_file() for test_file in represented_test_files) diff --git a/tests/workflow_cross_field/SUMMARY.md b/tests/workflow_cross_field/SUMMARY.md new file mode 100644 index 0000000..34900f9 --- /dev/null +++ b/tests/workflow_cross_field/SUMMARY.md @@ -0,0 +1,46 @@ +# Cross-Field Coverage Summary + +Generated deterministically by `tests.workflow_cross_field.artifacts`. + +Schema version: 1. + +| Product law | model | real_route | real_coordinator | browser_smoke | +|---|---|---|---|---| +| `api.hosted_desktop_only_routes_unavailable` | uncovered | uncovered | uncovered | passed | +| `assistant.disable_clears_live_pack` | passed | uncovered | uncovered | uncovered | +| `assistant.no_validator_injection` | passed | uncovered | uncovered | uncovered | +| `assistant.non_blocking_retrieval` | uncovered | uncovered | uncovered | uncovered | +| `assistant.stagnant_pack_backoff_not_shutdown` | uncovered | uncovered | uncovered | uncovered | +| `events.context_overflow_persists_across_reload` | passed | uncovered | passed | passed | +| `events.context_overflow_route_identity` | passed | uncovered | passed | passed | +| `events.context_overflow_terminal_stop_once` | passed | uncovered | passed | passed | +| `events.frontend_events_include_scope_phase` | passed | uncovered | uncovered | uncovered | +| `events.proof_context_overflow_nonfatal` | passed | uncovered | passed | passed | +| `events.proof_verified_after_registration` | passed | uncovered | blocked (The production final loop combines unbounded model iteration, Lean execution, semantic review, persistence, and registration without a bounded route-level seam. Direct registration ordering is covered elsewhere; publishing a full lifecycle pass would require faking the transition owner rather than observing it.) | uncovered | +| `outputs.at_least_one_output_enabled` | passed | passed | uncovered | passed | +| `outputs.papers_only_skips_proof_work` | passed | uncovered | uncovered | uncovered | +| `outputs.proofs_only_no_paper_phase` | passed | uncovered | passed | uncovered | +| `prompt.direct_sources_excluded_from_rag` | passed | uncovered | passed | uncovered | +| `prompt.generated_appendices_stripped` | passed | uncovered | passed | uncovered | +| `prompt.mandatory_source_overflow_fails_visible` | passed | uncovered | passed | uncovered | +| `prompt.proof_source_context_required` | passed | uncovered | uncovered | uncovered | +| `prompt.user_prompt_direct_injected` | passed | uncovered | uncovered | uncovered | +| `prompt.validator_excludes_assistant_memory` | passed | uncovered | uncovered | uncovered | +| `proof_runtime.hosted_proof_settings_unavailable` | uncovered | passed | uncovered | uncovered | +| `proof_runtime.no_lean_when_disabled` | passed | uncovered | passed | uncovered | +| `proof_runtime.no_smt_when_disabled` | passed | uncovered | uncovered | uncovered | +| `proof_runtime.truncation_is_attempt_failure` | uncovered | uncovered | uncovered | uncovered | +| `proof_scope.autonomous_events_not_manual` | passed | uncovered | passed | uncovered | +| `proof_scope.generated_appendices_stripped_from_prompts` | uncovered | uncovered | passed | uncovered | +| `proof_scope.manual_clear_archives_active` | passed | passed | passed | uncovered | +| `proof_scope.manual_not_in_autonomous_current` | passed | passed | passed | uncovered | +| `provider.pause_preserves_checkpoint` | passed | uncovered | passed | uncovered | +| `provider.reset_wakes_without_corrupting_checkpoint` | passed | uncovered | uncovered | uncovered | +| `provider.stop_resume_preserves_pause` | passed | uncovered | blocked (Existing coordinator coverage observes pause/reset resume, but no bounded test seam independently interleaves user Stop while the provider wait is active.) | uncovered | +| `runtime.child_tasks_count_as_active` | uncovered | passed | uncovered | uncovered | +| `runtime.parent_action_fences_child_outputs` | uncovered | uncovered | blocked (The invariant catalog declares model-only support and no existing isolated production seam exposes stale child output acceptance after parent takeover.) | uncovered | +| `runtime.single_active_workflow` | passed | passed | uncovered | passed | +| `state.clear_removes_active_preserves_history` | passed | passed | passed | uncovered | +| `state.completed_brainstorm_no_replay_handoff` | uncovered | uncovered | uncovered | uncovered | +| `state.pruned_papers_excluded_from_context` | uncovered | passed | uncovered | uncovered | +| `state.runtime_roots_are_active_roots` | uncovered | uncovered | passed | uncovered | diff --git a/tests/workflow_cross_field/SUPPORT_GRAPH.md b/tests/workflow_cross_field/SUPPORT_GRAPH.md new file mode 100644 index 0000000..1671e3a --- /dev/null +++ b/tests/workflow_cross_field/SUPPORT_GRAPH.md @@ -0,0 +1,185 @@ +# Workflow Support Graph + +Generated deterministically from separately versioned declared and observed support bases. + +Schema version: 1. + +| Risk | Invariant | Field | Adapter | Status | Basis | Scenario / reason | +|---|---|---|---|---|---|---| +| `risk.assistant_memory_crosses_boundary` | `assistant.disable_clears_live_pack` | `assistant_memory` | `browser_smoke` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.assistant_memory_crosses_boundary` | `assistant.disable_clears_live_pack` | `assistant_memory` | `model` | passed | declared=yes, observed=yes | generated_assistant_prompt_context_seed_29 | +| `risk.assistant_memory_crosses_boundary` | `assistant.disable_clears_live_pack` | `assistant_memory` | `model` | passed | declared=yes, observed=yes | generated_assistant_prompt_context_seed_7 | +| `risk.assistant_memory_crosses_boundary` | `assistant.disable_clears_live_pack` | `assistant_memory` | `model` | passed | declared=yes, observed=yes | model_assistant_prompt_validator_exclusion | +| `risk.assistant_memory_crosses_boundary` | `assistant.disable_clears_live_pack` | `assistant_memory` | `model` | passed | declared=yes, observed=yes | model_manual_proof_clear_scope_assistant | +| `risk.assistant_memory_crosses_boundary` | `assistant.disable_clears_live_pack` | `assistant_memory` | `real_coordinator` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.assistant_memory_crosses_boundary` | `assistant.disable_clears_live_pack` | `assistant_memory` | `real_route` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.assistant_memory_crosses_boundary` | `assistant.no_validator_injection` | `assistant_memory` | `browser_smoke` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.assistant_memory_crosses_boundary` | `assistant.no_validator_injection` | `assistant_memory` | `model` | passed | declared=yes, observed=yes | generated_assistant_prompt_context_seed_29 | +| `risk.assistant_memory_crosses_boundary` | `assistant.no_validator_injection` | `assistant_memory` | `model` | passed | declared=yes, observed=yes | generated_assistant_prompt_context_seed_7 | +| `risk.assistant_memory_crosses_boundary` | `assistant.no_validator_injection` | `assistant_memory` | `model` | passed | declared=yes, observed=yes | model_assistant_prompt_validator_exclusion | +| `risk.assistant_memory_crosses_boundary` | `assistant.no_validator_injection` | `assistant_memory` | `real_coordinator` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.assistant_memory_crosses_boundary` | `assistant.no_validator_injection` | `assistant_memory` | `real_route` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.assistant_memory_crosses_boundary` | `prompt.validator_excludes_assistant_memory` | `prompt_context` | `browser_smoke` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.assistant_memory_crosses_boundary` | `prompt.validator_excludes_assistant_memory` | `prompt_context` | `model` | passed | declared=yes, observed=yes | generated_assistant_prompt_context_seed_29 | +| `risk.assistant_memory_crosses_boundary` | `prompt.validator_excludes_assistant_memory` | `prompt_context` | `model` | passed | declared=yes, observed=yes | generated_assistant_prompt_context_seed_7 | +| `risk.assistant_memory_crosses_boundary` | `prompt.validator_excludes_assistant_memory` | `prompt_context` | `model` | passed | declared=yes, observed=yes | model_assistant_prompt_validator_exclusion | +| `risk.assistant_memory_crosses_boundary` | `prompt.validator_excludes_assistant_memory` | `prompt_context` | `real_coordinator` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.assistant_memory_crosses_boundary` | `prompt.validator_excludes_assistant_memory` | `prompt_context` | `real_route` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.clear_destroys_history` | `proof_scope.manual_clear_archives_active` | `proof_scope_isolation` | `browser_smoke` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.clear_destroys_history` | `proof_scope.manual_clear_archives_active` | `proof_scope_isolation` | `model` | passed | declared=yes, observed=yes | generated_manual_scope_clear_archive_seed_29 | +| `risk.clear_destroys_history` | `proof_scope.manual_clear_archives_active` | `proof_scope_isolation` | `model` | passed | declared=yes, observed=yes | generated_manual_scope_clear_archive_seed_7 | +| `risk.clear_destroys_history` | `proof_scope.manual_clear_archives_active` | `proof_scope_isolation` | `model` | passed | declared=yes, observed=yes | model_manual_proof_clear_scope_assistant | +| `risk.clear_destroys_history` | `proof_scope.manual_clear_archives_active` | `proof_scope_isolation` | `real_coordinator` | passed | declared=yes, observed=yes | real_manual_archive_and_leanoj_clear_preserve_scope_contract | +| `risk.clear_destroys_history` | `proof_scope.manual_clear_archives_active` | `proof_scope_isolation` | `real_route` | passed | declared=yes, observed=yes | real_manual_aggregator_clear_archives_before_clear | +| `risk.clear_destroys_history` | `proof_scope.manual_clear_archives_active` | `proof_scope_isolation` | `real_route` | passed | declared=yes, observed=yes | real_manual_compiler_papers_lifecycle_and_clear_archive | +| `risk.clear_destroys_history` | `state.clear_removes_active_preserves_history` | `workflow_filesystem_state` | `browser_smoke` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.clear_destroys_history` | `state.clear_removes_active_preserves_history` | `workflow_filesystem_state` | `model` | passed | declared=yes, observed=yes | generated_manual_scope_clear_archive_seed_29 | +| `risk.clear_destroys_history` | `state.clear_removes_active_preserves_history` | `workflow_filesystem_state` | `model` | passed | declared=yes, observed=yes | generated_manual_scope_clear_archive_seed_7 | +| `risk.clear_destroys_history` | `state.clear_removes_active_preserves_history` | `workflow_filesystem_state` | `model` | passed | declared=yes, observed=yes | model_manual_proof_clear_scope_assistant | +| `risk.clear_destroys_history` | `state.clear_removes_active_preserves_history` | `workflow_filesystem_state` | `real_coordinator` | passed | declared=yes, observed=yes | real_manual_archive_and_leanoj_clear_preserve_scope_contract | +| `risk.clear_destroys_history` | `state.clear_removes_active_preserves_history` | `workflow_filesystem_state` | `real_route` | passed | declared=yes, observed=yes | real_manual_compiler_papers_lifecycle_and_clear_archive | +| `risk.context_overflow_activity_loses_identity` | `events.context_overflow_persists_across_reload` | `websocket_api_contracts` | `browser_smoke` | passed | declared=yes, observed=yes | browser_overflow_activity_persists_with_attribution_and_terminal_deduplication | +| `risk.context_overflow_activity_loses_identity` | `events.context_overflow_persists_across_reload` | `websocket_api_contracts` | `model` | passed | declared=yes, observed=yes | generated_context_overflow_contract_seed_29 | +| `risk.context_overflow_activity_loses_identity` | `events.context_overflow_persists_across_reload` | `websocket_api_contracts` | `model` | passed | declared=yes, observed=yes | generated_context_overflow_contract_seed_7 | +| `risk.context_overflow_activity_loses_identity` | `events.context_overflow_persists_across_reload` | `websocket_api_contracts` | `real_coordinator` | passed | declared=yes, observed=yes | real_manual_aggregator_overflow_identity_persists_across_reload | +| `risk.context_overflow_activity_loses_identity` | `events.context_overflow_persists_across_reload` | `websocket_api_contracts` | `real_route` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.context_overflow_activity_loses_identity` | `events.context_overflow_route_identity` | `websocket_api_contracts` | `browser_smoke` | passed | declared=yes, observed=yes | browser_overflow_activity_persists_with_attribution_and_terminal_deduplication | +| `risk.context_overflow_activity_loses_identity` | `events.context_overflow_route_identity` | `websocket_api_contracts` | `model` | passed | declared=yes, observed=yes | generated_context_overflow_contract_seed_29 | +| `risk.context_overflow_activity_loses_identity` | `events.context_overflow_route_identity` | `websocket_api_contracts` | `model` | passed | declared=yes, observed=yes | generated_context_overflow_contract_seed_7 | +| `risk.context_overflow_activity_loses_identity` | `events.context_overflow_route_identity` | `websocket_api_contracts` | `real_coordinator` | passed | declared=yes, observed=yes | real_autonomous_overflow_terminal_stop_once | +| `risk.context_overflow_activity_loses_identity` | `events.context_overflow_route_identity` | `websocket_api_contracts` | `real_coordinator` | passed | declared=yes, observed=yes | real_manual_aggregator_overflow_identity_persists_across_reload | +| `risk.context_overflow_activity_loses_identity` | `events.context_overflow_route_identity` | `websocket_api_contracts` | `real_coordinator` | passed | declared=yes, observed=yes | real_proof_context_overflow_is_scoped_nonfatal | +| `risk.context_overflow_activity_loses_identity` | `events.context_overflow_route_identity` | `websocket_api_contracts` | `real_route` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.context_overflow_activity_loses_identity` | `events.context_overflow_terminal_stop_once` | `websocket_api_contracts` | `browser_smoke` | passed | declared=yes, observed=yes | browser_overflow_activity_persists_with_attribution_and_terminal_deduplication | +| `risk.context_overflow_activity_loses_identity` | `events.context_overflow_terminal_stop_once` | `websocket_api_contracts` | `model` | passed | declared=yes, observed=yes | generated_context_overflow_contract_seed_29 | +| `risk.context_overflow_activity_loses_identity` | `events.context_overflow_terminal_stop_once` | `websocket_api_contracts` | `model` | passed | declared=yes, observed=yes | generated_context_overflow_contract_seed_7 | +| `risk.context_overflow_activity_loses_identity` | `events.context_overflow_terminal_stop_once` | `websocket_api_contracts` | `real_coordinator` | passed | declared=yes, observed=yes | real_autonomous_overflow_terminal_stop_once | +| `risk.context_overflow_activity_loses_identity` | `events.context_overflow_terminal_stop_once` | `websocket_api_contracts` | `real_route` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.context_overflow_activity_loses_identity` | `events.proof_context_overflow_nonfatal` | `websocket_api_contracts` | `browser_smoke` | passed | declared=yes, observed=yes | browser_overflow_activity_persists_with_attribution_and_terminal_deduplication | +| `risk.context_overflow_activity_loses_identity` | `events.proof_context_overflow_nonfatal` | `websocket_api_contracts` | `model` | passed | declared=yes, observed=yes | generated_context_overflow_contract_seed_29 | +| `risk.context_overflow_activity_loses_identity` | `events.proof_context_overflow_nonfatal` | `websocket_api_contracts` | `model` | passed | declared=yes, observed=yes | generated_context_overflow_contract_seed_7 | +| `risk.context_overflow_activity_loses_identity` | `events.proof_context_overflow_nonfatal` | `websocket_api_contracts` | `real_coordinator` | passed | declared=yes, observed=yes | real_proof_context_overflow_is_scoped_nonfatal | +| `risk.context_overflow_activity_loses_identity` | `events.proof_context_overflow_nonfatal` | `websocket_api_contracts` | `real_route` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.direct_source_duplicated_by_rag` | `prompt.direct_sources_excluded_from_rag` | `prompt_context` | `browser_smoke` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.direct_source_duplicated_by_rag` | `prompt.direct_sources_excluded_from_rag` | `prompt_context` | `model` | passed | declared=yes, observed=yes | generated_rag_prompt_boundaries_seed_29 | +| `risk.direct_source_duplicated_by_rag` | `prompt.direct_sources_excluded_from_rag` | `prompt_context` | `model` | passed | declared=yes, observed=yes | generated_rag_prompt_boundaries_seed_7 | +| `risk.direct_source_duplicated_by_rag` | `prompt.direct_sources_excluded_from_rag` | `prompt_context` | `real_coordinator` | passed | declared=yes, observed=yes | real_direct_source_rag_exclusion_matrix | +| `risk.direct_source_duplicated_by_rag` | `prompt.direct_sources_excluded_from_rag` | `prompt_context` | `real_route` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.direct_source_duplicated_by_rag` | `prompt.mandatory_source_overflow_fails_visible` | `prompt_context` | `browser_smoke` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.direct_source_duplicated_by_rag` | `prompt.mandatory_source_overflow_fails_visible` | `prompt_context` | `model` | passed | declared=yes, observed=yes | generated_rag_prompt_boundaries_seed_29 | +| `risk.direct_source_duplicated_by_rag` | `prompt.mandatory_source_overflow_fails_visible` | `prompt_context` | `model` | passed | declared=yes, observed=yes | generated_rag_prompt_boundaries_seed_7 | +| `risk.direct_source_duplicated_by_rag` | `prompt.mandatory_source_overflow_fails_visible` | `prompt_context` | `real_coordinator` | passed | declared=yes, observed=yes | real_rigor_mandatory_source_overflow_fails_before_model_or_rag | +| `risk.direct_source_duplicated_by_rag` | `prompt.mandatory_source_overflow_fails_visible` | `prompt_context` | `real_route` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.disabled_proof_runtime_executes` | `proof_runtime.no_lean_when_disabled` | `proof_runtime_gating` | `browser_smoke` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.disabled_proof_runtime_executes` | `proof_runtime.no_lean_when_disabled` | `proof_runtime_gating` | `model` | passed | declared=yes, observed=yes | generated_output_and_proof_runtime_gating_seed_29 | +| `risk.disabled_proof_runtime_executes` | `proof_runtime.no_lean_when_disabled` | `proof_runtime_gating` | `model` | passed | declared=yes, observed=yes | generated_output_and_proof_runtime_gating_seed_7 | +| `risk.disabled_proof_runtime_executes` | `proof_runtime.no_lean_when_disabled` | `proof_runtime_gating` | `model` | passed | declared=yes, observed=yes | model_proof_runtime_gating_allowed_outputs | +| `risk.disabled_proof_runtime_executes` | `proof_runtime.no_lean_when_disabled` | `proof_runtime_gating` | `real_coordinator` | passed | declared=yes, observed=yes | real_disabled_proof_runtime_skips_autonomous_checkpoint | +| `risk.disabled_proof_runtime_executes` | `proof_runtime.no_lean_when_disabled` | `proof_runtime_gating` | `real_route` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.disabled_proof_runtime_executes` | `proof_runtime.no_smt_when_disabled` | `proof_runtime_gating` | `browser_smoke` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.disabled_proof_runtime_executes` | `proof_runtime.no_smt_when_disabled` | `proof_runtime_gating` | `model` | passed | declared=yes, observed=yes | generated_output_and_proof_runtime_gating_seed_29 | +| `risk.disabled_proof_runtime_executes` | `proof_runtime.no_smt_when_disabled` | `proof_runtime_gating` | `model` | passed | declared=yes, observed=yes | generated_output_and_proof_runtime_gating_seed_7 | +| `risk.disabled_proof_runtime_executes` | `proof_runtime.no_smt_when_disabled` | `proof_runtime_gating` | `model` | passed | declared=yes, observed=yes | model_proof_runtime_gating_allowed_outputs | +| `risk.disabled_proof_runtime_executes` | `proof_runtime.no_smt_when_disabled` | `proof_runtime_gating` | `real_coordinator` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.disabled_proof_runtime_executes` | `proof_runtime.no_smt_when_disabled` | `proof_runtime_gating` | `real_route` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.generated_proofs_reenter_prompts` | `prompt.generated_appendices_stripped` | `prompt_context` | `browser_smoke` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.generated_proofs_reenter_prompts` | `prompt.generated_appendices_stripped` | `prompt_context` | `model` | passed | declared=yes, observed=yes | generated_rag_prompt_boundaries_seed_29 | +| `risk.generated_proofs_reenter_prompts` | `prompt.generated_appendices_stripped` | `prompt_context` | `model` | passed | declared=yes, observed=yes | generated_rag_prompt_boundaries_seed_7 | +| `risk.generated_proofs_reenter_prompts` | `prompt.generated_appendices_stripped` | `prompt_context` | `real_coordinator` | passed | declared=yes, observed=yes | real_model_visible_prompts_strip_generated_proof_appendices | +| `risk.generated_proofs_reenter_prompts` | `prompt.generated_appendices_stripped` | `prompt_context` | `real_route` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.generated_proofs_reenter_prompts` | `prompt.proof_source_context_required` | `prompt_context` | `browser_smoke` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.generated_proofs_reenter_prompts` | `prompt.proof_source_context_required` | `prompt_context` | `model` | passed | declared=yes, observed=yes | generated_rag_prompt_boundaries_seed_29 | +| `risk.generated_proofs_reenter_prompts` | `prompt.proof_source_context_required` | `prompt_context` | `model` | passed | declared=yes, observed=yes | generated_rag_prompt_boundaries_seed_7 | +| `risk.generated_proofs_reenter_prompts` | `prompt.proof_source_context_required` | `prompt_context` | `real_coordinator` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.generated_proofs_reenter_prompts` | `prompt.proof_source_context_required` | `prompt_context` | `real_route` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.generated_proofs_reenter_prompts` | `proof_scope.generated_appendices_stripped_from_prompts` | `proof_scope_isolation` | `browser_smoke` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.generated_proofs_reenter_prompts` | `proof_scope.generated_appendices_stripped_from_prompts` | `proof_scope_isolation` | `model` | uncovered | declared=yes, observed=no | no observed scenario | +| `risk.generated_proofs_reenter_prompts` | `proof_scope.generated_appendices_stripped_from_prompts` | `proof_scope_isolation` | `real_coordinator` | passed | declared=yes, observed=yes | real_generated_proof_appendix_stripping | +| `risk.generated_proofs_reenter_prompts` | `proof_scope.generated_appendices_stripped_from_prompts` | `proof_scope_isolation` | `real_coordinator` | passed | declared=yes, observed=yes | real_model_visible_prompts_strip_generated_proof_appendices | +| `risk.generated_proofs_reenter_prompts` | `proof_scope.generated_appendices_stripped_from_prompts` | `proof_scope_isolation` | `real_route` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.hosted_desktop_boundary` | `api.hosted_desktop_only_routes_unavailable` | `websocket_api_contracts` | `browser_smoke` | passed | declared=yes, observed=yes | browser_hosted_startup_respects_desktop_capability_boundaries | +| `risk.hosted_desktop_boundary` | `api.hosted_desktop_only_routes_unavailable` | `websocket_api_contracts` | `model` | uncovered | declared=yes, observed=no | no observed scenario | +| `risk.hosted_desktop_boundary` | `api.hosted_desktop_only_routes_unavailable` | `websocket_api_contracts` | `real_coordinator` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.hosted_desktop_boundary` | `api.hosted_desktop_only_routes_unavailable` | `websocket_api_contracts` | `real_route` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.hosted_desktop_boundary` | `proof_runtime.hosted_proof_settings_unavailable` | `proof_runtime_gating` | `browser_smoke` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.hosted_desktop_boundary` | `proof_runtime.hosted_proof_settings_unavailable` | `proof_runtime_gating` | `model` | uncovered | declared=yes, observed=no | no observed scenario | +| `risk.hosted_desktop_boundary` | `proof_runtime.hosted_proof_settings_unavailable` | `proof_runtime_gating` | `real_coordinator` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.hosted_desktop_boundary` | `proof_runtime.hosted_proof_settings_unavailable` | `proof_runtime_gating` | `real_route` | passed | declared=yes, observed=yes | real_hosted_proof_settings_returns_desktop_unavailable | +| `risk.proof_scope_leak` | `proof_scope.autonomous_events_not_manual` | `proof_scope_isolation` | `browser_smoke` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.proof_scope_leak` | `proof_scope.autonomous_events_not_manual` | `proof_scope_isolation` | `model` | passed | declared=yes, observed=yes | generated_scoped_registered_proof_events_seed_29 | +| `risk.proof_scope_leak` | `proof_scope.autonomous_events_not_manual` | `proof_scope_isolation` | `model` | passed | declared=yes, observed=yes | generated_scoped_registered_proof_events_seed_7 | +| `risk.proof_scope_leak` | `proof_scope.autonomous_events_not_manual` | `proof_scope_isolation` | `model` | passed | declared=yes, observed=yes | model_websocket_proof_scope_events | +| `risk.proof_scope_leak` | `proof_scope.autonomous_events_not_manual` | `proof_scope_isolation` | `real_coordinator` | passed | declared=yes, observed=yes | real_leanoj_master_proof_stop_resume_preserves_isolated_state | +| `risk.proof_scope_leak` | `proof_scope.autonomous_events_not_manual` | `proof_scope_isolation` | `real_route` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.proof_scope_leak` | `proof_scope.manual_clear_archives_active` | `proof_scope_isolation` | `browser_smoke` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.proof_scope_leak` | `proof_scope.manual_clear_archives_active` | `proof_scope_isolation` | `model` | passed | declared=yes, observed=yes | generated_manual_scope_clear_archive_seed_29 | +| `risk.proof_scope_leak` | `proof_scope.manual_clear_archives_active` | `proof_scope_isolation` | `model` | passed | declared=yes, observed=yes | generated_manual_scope_clear_archive_seed_7 | +| `risk.proof_scope_leak` | `proof_scope.manual_clear_archives_active` | `proof_scope_isolation` | `model` | passed | declared=yes, observed=yes | model_manual_proof_clear_scope_assistant | +| `risk.proof_scope_leak` | `proof_scope.manual_clear_archives_active` | `proof_scope_isolation` | `real_coordinator` | passed | declared=yes, observed=yes | real_manual_archive_and_leanoj_clear_preserve_scope_contract | +| `risk.proof_scope_leak` | `proof_scope.manual_clear_archives_active` | `proof_scope_isolation` | `real_route` | passed | declared=yes, observed=yes | real_manual_aggregator_clear_archives_before_clear | +| `risk.proof_scope_leak` | `proof_scope.manual_clear_archives_active` | `proof_scope_isolation` | `real_route` | passed | declared=yes, observed=yes | real_manual_compiler_papers_lifecycle_and_clear_archive | +| `risk.proof_scope_leak` | `proof_scope.manual_not_in_autonomous_current` | `proof_scope_isolation` | `browser_smoke` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.proof_scope_leak` | `proof_scope.manual_not_in_autonomous_current` | `proof_scope_isolation` | `model` | passed | declared=yes, observed=yes | generated_manual_scope_clear_archive_seed_29 | +| `risk.proof_scope_leak` | `proof_scope.manual_not_in_autonomous_current` | `proof_scope_isolation` | `model` | passed | declared=yes, observed=yes | generated_manual_scope_clear_archive_seed_7 | +| `risk.proof_scope_leak` | `proof_scope.manual_not_in_autonomous_current` | `proof_scope_isolation` | `model` | passed | declared=yes, observed=yes | model_manual_proof_clear_scope_assistant | +| `risk.proof_scope_leak` | `proof_scope.manual_not_in_autonomous_current` | `proof_scope_isolation` | `real_coordinator` | passed | declared=yes, observed=yes | real_manual_compiler_proof_only_uses_rigor_settings_and_manual_scope | +| `risk.proof_scope_leak` | `proof_scope.manual_not_in_autonomous_current` | `proof_scope_isolation` | `real_route` | passed | declared=yes, observed=yes | real_proof_scope_matrix_isolated_under_temp_root | +| `risk.proofs_only_enters_paper` | `outputs.at_least_one_output_enabled` | `allowed_outputs` | `browser_smoke` | passed | declared=yes, observed=yes | browser_autonomous_start_stop_preserves_single_ui_owner | +| `risk.proofs_only_enters_paper` | `outputs.at_least_one_output_enabled` | `allowed_outputs` | `model` | passed | declared=yes, observed=yes | generated_output_and_proof_runtime_gating_seed_29 | +| `risk.proofs_only_enters_paper` | `outputs.at_least_one_output_enabled` | `allowed_outputs` | `model` | passed | declared=yes, observed=yes | generated_output_and_proof_runtime_gating_seed_7 | +| `risk.proofs_only_enters_paper` | `outputs.at_least_one_output_enabled` | `allowed_outputs` | `model` | passed | declared=yes, observed=yes | model_proof_runtime_gating_allowed_outputs | +| `risk.proofs_only_enters_paper` | `outputs.at_least_one_output_enabled` | `allowed_outputs` | `real_coordinator` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.proofs_only_enters_paper` | `outputs.at_least_one_output_enabled` | `allowed_outputs` | `real_route` | passed | declared=yes, observed=yes | real_manual_compiler_rejects_no_allowed_outputs | +| `risk.proofs_only_enters_paper` | `outputs.proofs_only_no_paper_phase` | `allowed_outputs` | `browser_smoke` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.proofs_only_enters_paper` | `outputs.proofs_only_no_paper_phase` | `allowed_outputs` | `model` | passed | declared=yes, observed=yes | generated_allowed_outputs_provider_pause_resume_seed_29 | +| `risk.proofs_only_enters_paper` | `outputs.proofs_only_no_paper_phase` | `allowed_outputs` | `model` | passed | declared=yes, observed=yes | generated_allowed_outputs_provider_pause_resume_seed_7 | +| `risk.proofs_only_enters_paper` | `outputs.proofs_only_no_paper_phase` | `allowed_outputs` | `model` | passed | declared=yes, observed=yes | model_allowed_outputs_provider_pause_stop_resume | +| `risk.proofs_only_enters_paper` | `outputs.proofs_only_no_paper_phase` | `allowed_outputs` | `real_coordinator` | passed | declared=yes, observed=yes | real_autonomous_proofs_only_handoff_returns_to_topic_exploration | +| `risk.proofs_only_enters_paper` | `outputs.proofs_only_no_paper_phase` | `allowed_outputs` | `real_route` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.provider_pause_loses_checkpoint` | `provider.pause_preserves_checkpoint` | `provider_pause_resume` | `browser_smoke` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.provider_pause_loses_checkpoint` | `provider.pause_preserves_checkpoint` | `provider_pause_resume` | `model` | passed | declared=yes, observed=yes | generated_allowed_outputs_provider_pause_resume_seed_29 | +| `risk.provider_pause_loses_checkpoint` | `provider.pause_preserves_checkpoint` | `provider_pause_resume` | `model` | passed | declared=yes, observed=yes | generated_allowed_outputs_provider_pause_resume_seed_7 | +| `risk.provider_pause_loses_checkpoint` | `provider.pause_preserves_checkpoint` | `provider_pause_resume` | `model` | passed | declared=yes, observed=yes | model_allowed_outputs_provider_pause_stop_resume | +| `risk.provider_pause_loses_checkpoint` | `provider.pause_preserves_checkpoint` | `provider_pause_resume` | `real_coordinator` | passed | declared=yes, observed=yes | real_autonomous_provider_credit_pause_preserves_checkpoint_and_resumes | +| `risk.provider_pause_loses_checkpoint` | `provider.pause_preserves_checkpoint` | `provider_pause_resume` | `real_route` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.provider_pause_loses_checkpoint` | `provider.reset_wakes_without_corrupting_checkpoint` | `provider_pause_resume` | `browser_smoke` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.provider_pause_loses_checkpoint` | `provider.reset_wakes_without_corrupting_checkpoint` | `provider_pause_resume` | `model` | passed | declared=yes, observed=yes | generated_allowed_outputs_provider_pause_resume_seed_29 | +| `risk.provider_pause_loses_checkpoint` | `provider.reset_wakes_without_corrupting_checkpoint` | `provider_pause_resume` | `model` | passed | declared=yes, observed=yes | generated_allowed_outputs_provider_pause_resume_seed_7 | +| `risk.provider_pause_loses_checkpoint` | `provider.reset_wakes_without_corrupting_checkpoint` | `provider_pause_resume` | `model` | passed | declared=yes, observed=yes | model_allowed_outputs_provider_pause_stop_resume | +| `risk.provider_pause_loses_checkpoint` | `provider.reset_wakes_without_corrupting_checkpoint` | `provider_pause_resume` | `real_coordinator` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.provider_pause_loses_checkpoint` | `provider.reset_wakes_without_corrupting_checkpoint` | `provider_pause_resume` | `real_route` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.provider_pause_loses_checkpoint` | `provider.stop_resume_preserves_pause` | `provider_pause_resume` | `browser_smoke` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.provider_pause_loses_checkpoint` | `provider.stop_resume_preserves_pause` | `provider_pause_resume` | `model` | passed | declared=yes, observed=yes | generated_allowed_outputs_provider_pause_resume_seed_29 | +| `risk.provider_pause_loses_checkpoint` | `provider.stop_resume_preserves_pause` | `provider_pause_resume` | `model` | passed | declared=yes, observed=yes | generated_allowed_outputs_provider_pause_resume_seed_7 | +| `risk.provider_pause_loses_checkpoint` | `provider.stop_resume_preserves_pause` | `provider_pause_resume` | `model` | passed | declared=yes, observed=yes | generated_leanoj_durable_lifecycle_seed_29 | +| `risk.provider_pause_loses_checkpoint` | `provider.stop_resume_preserves_pause` | `provider_pause_resume` | `model` | passed | declared=yes, observed=yes | generated_leanoj_durable_lifecycle_seed_7 | +| `risk.provider_pause_loses_checkpoint` | `provider.stop_resume_preserves_pause` | `provider_pause_resume` | `model` | passed | declared=yes, observed=yes | model_allowed_outputs_provider_pause_stop_resume | +| `risk.provider_pause_loses_checkpoint` | `provider.stop_resume_preserves_pause` | `provider_pause_resume` | `real_coordinator` | blocked | declared=no, observed=yes | real_provider_stop_reset_checkpoint_unavailable_without_wait_seam | +| `risk.provider_pause_loses_checkpoint` | `provider.stop_resume_preserves_pause` | `provider_pause_resume` | `real_route` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.pruned_paper_context_leak` | `state.pruned_papers_excluded_from_context` | `workflow_filesystem_state` | `browser_smoke` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.pruned_paper_context_leak` | `state.pruned_papers_excluded_from_context` | `workflow_filesystem_state` | `model` | uncovered | declared=yes, observed=no | no observed scenario | +| `risk.pruned_paper_context_leak` | `state.pruned_papers_excluded_from_context` | `workflow_filesystem_state` | `real_coordinator` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.pruned_paper_context_leak` | `state.pruned_papers_excluded_from_context` | `workflow_filesystem_state` | `real_route` | passed | declared=yes, observed=yes | real_pruned_paper_preserved_but_removed_from_active_context | +| `risk.stale_child_phase_takeover` | `runtime.parent_action_fences_child_outputs` | `runtime_exclusivity` | `browser_smoke` | uncovered | declared=no, observed=no | no observed scenario | +| `risk.stale_child_phase_takeover` | `runtime.parent_action_fences_child_outputs` | `runtime_exclusivity` | `model` | uncovered | declared=yes, observed=no | no observed scenario | +| `risk.stale_child_phase_takeover` | `runtime.parent_action_fences_child_outputs` | `runtime_exclusivity` | `real_coordinator` | blocked | declared=no, observed=yes | real_parent_action_fencing_unavailable_without_production_seam | +| `risk.stale_child_phase_takeover` | `runtime.parent_action_fences_child_outputs` | `runtime_exclusivity` | `real_route` | uncovered | declared=no, observed=no | no observed scenario | + +## Structured Gaps + +| Gap | Status | Declared | Reason | +|---|---|---|---| +| `risk.assistant_memory_crosses_boundary:assistant.disable_clears_live_pack:real_coordinator` | uncovered | no | No observed coverage record exists for this required risk/invariant/adapter basis. | +| `risk.assistant_memory_crosses_boundary:assistant.no_validator_injection:real_coordinator` | uncovered | no | No observed coverage record exists for this required risk/invariant/adapter basis. | +| `risk.assistant_memory_crosses_boundary:prompt.validator_excludes_assistant_memory:real_coordinator` | uncovered | no | No observed coverage record exists for this required risk/invariant/adapter basis. | +| `risk.disabled_proof_runtime_executes:proof_runtime.no_smt_when_disabled:real_coordinator` | uncovered | no | No observed coverage record exists for this required risk/invariant/adapter basis. | +| `risk.generated_proofs_reenter_prompts:prompt.proof_source_context_required:real_coordinator` | uncovered | no | No observed coverage record exists for this required risk/invariant/adapter basis. | +| `risk.generated_proofs_reenter_prompts:proof_scope.generated_appendices_stripped_from_prompts:model` | uncovered | yes | No observed coverage record exists for this required risk/invariant/adapter basis. | +| `risk.hosted_desktop_boundary:api.hosted_desktop_only_routes_unavailable:model` | uncovered | yes | No observed coverage record exists for this required risk/invariant/adapter basis. | +| `risk.hosted_desktop_boundary:api.hosted_desktop_only_routes_unavailable:real_route` | uncovered | no | No observed coverage record exists for this required risk/invariant/adapter basis. | +| `risk.hosted_desktop_boundary:proof_runtime.hosted_proof_settings_unavailable:model` | uncovered | yes | No observed coverage record exists for this required risk/invariant/adapter basis. | +| `risk.proofs_only_enters_paper:outputs.at_least_one_output_enabled:real_coordinator` | uncovered | no | No observed coverage record exists for this required risk/invariant/adapter basis. | +| `risk.provider_pause_loses_checkpoint:provider.reset_wakes_without_corrupting_checkpoint:real_coordinator` | uncovered | no | No observed coverage record exists for this required risk/invariant/adapter basis. | +| `risk.pruned_paper_context_leak:state.pruned_papers_excluded_from_context:model` | uncovered | yes | No observed coverage record exists for this required risk/invariant/adapter basis. | +| `risk.pruned_paper_context_leak:state.pruned_papers_excluded_from_context:real_coordinator` | uncovered | no | No observed coverage record exists for this required risk/invariant/adapter basis. | +| `risk.stale_child_phase_takeover:runtime.parent_action_fences_child_outputs:model` | uncovered | yes | No observed coverage record exists for this required risk/invariant/adapter basis. | diff --git a/tests/workflow_cross_field/TARGET_RISKS.md b/tests/workflow_cross_field/TARGET_RISKS.md new file mode 100644 index 0000000..68b944f --- /dev/null +++ b/tests/workflow_cross_field/TARGET_RISKS.md @@ -0,0 +1,20 @@ +# Inverse Target-Risk Analysis + +Generated deterministically by `tests.workflow_cross_field.artifacts` from Build A in-memory coverage records. + +Schema version: 1. + +| Rank | Target risk | Priority | Gap score | Adapter support | Missing isolated artifacts | +|---:|---|---|---:|---|---| +| 1 | `risk.hosted_desktop_boundary` — Hosted mode invokes desktop-only behavior | critical | 15 | model: uncovered; real_route: uncovered | `model:api.hosted_desktop_only_routes_unavailable`, `model:proof_runtime.hosted_proof_settings_unavailable`, `real_route:api.hosted_desktop_only_routes_unavailable` | +| 2 | `risk.stale_child_phase_takeover` — Stale child output overrides parent phase | critical | 12 | model: uncovered; real_coordinator: blocked | `model:runtime.parent_action_fences_child_outputs`, `real_coordinator:runtime.parent_action_fences_child_outputs` | +| 3 | `risk.provider_pause_loses_checkpoint` — Provider pause loses resumable work | critical | 9 | model: passed; real_coordinator: blocked | `real_coordinator:provider.reset_wakes_without_corrupting_checkpoint`, `real_coordinator:provider.stop_resume_preserves_pause` | +| 4 | `risk.assistant_memory_crosses_boundary` — Assistant memory crosses role boundary | high | 8 | model: passed; real_coordinator: uncovered | `real_coordinator:assistant.disable_clears_live_pack`, `real_coordinator:assistant.no_validator_injection`, `real_coordinator:prompt.validator_excludes_assistant_memory` | +| 5 | `risk.generated_proofs_reenter_prompts` — Generated proofs reenter source prompts | high | 8 | model: uncovered; real_coordinator: uncovered | `model:proof_scope.generated_appendices_stripped_from_prompts`, `real_coordinator:prompt.proof_source_context_required` | +| 6 | `risk.pruned_paper_context_leak` — Pruned paper remains in model context | high | 8 | model: uncovered; real_coordinator: uncovered | `model:state.pruned_papers_excluded_from_context`, `real_coordinator:state.pruned_papers_excluded_from_context` | +| 7 | `risk.disabled_proof_runtime_executes` — Disabled proof runtime executes tools | critical | 6 | model: passed; real_coordinator: uncovered | `real_coordinator:proof_runtime.no_smt_when_disabled` | +| 8 | `risk.proofs_only_enters_paper` — Proofs-only run enters paper phases | critical | 6 | model: passed; real_coordinator: uncovered | `real_coordinator:outputs.at_least_one_output_enabled` | +| 9 | `risk.clear_destroys_history` — Clear destroys history or leaves active state | high | 0 | model: passed; real_coordinator: passed | none | +| 10 | `risk.context_overflow_activity_loses_identity` — Context overflow loses route identity or lifecycle semantics | critical | 0 | model: passed; real_coordinator: passed | none | +| 11 | `risk.direct_source_duplicated_by_rag` — Direct source is duplicated or displaced by RAG | high | 0 | model: passed; real_coordinator: passed | none | +| 12 | `risk.proof_scope_leak` — Proof records leak across scopes | critical | 0 | model: passed; real_coordinator: passed | none | diff --git a/tests/workflow_cross_field/__init__.py b/tests/workflow_cross_field/__init__.py new file mode 100644 index 0000000..a0d7272 --- /dev/null +++ b/tests/workflow_cross_field/__init__.py @@ -0,0 +1 @@ +"""Deterministic, test-only workflow coverage artifacts.""" diff --git a/tests/workflow_cross_field/artifacts.py b/tests/workflow_cross_field/artifacts.py new file mode 100644 index 0000000..8428e0f --- /dev/null +++ b/tests/workflow_cross_field/artifacts.py @@ -0,0 +1,285 @@ +from __future__ import annotations + +import argparse +import json +import tempfile +from pathlib import Path + +from tests.workflow_harness.coverage_metadata import AdapterType, InteractionCoverage +from tests.workflow_harness.cross_field_scenarios import CROSS_FIELD_SCENARIOS +from tests.workflow_harness.invariant_catalog import INVARIANTS_BY_ID +from tests.workflow_harness.model import WorkflowModel +from tests.workflow_harness.recombinatory_generator import ( + DEFAULT_TARGETS, + run_generated_scenario, +) +from tests.workflow_browser_smoke.coverage_records import BROWSER_SMOKE_COVERAGE +from tests.workflow_real_adapters.coverage_records import REAL_ADAPTER_COVERAGE +from tests.workflow_cross_field.target_risks import ( + TARGET_RISK_SCHEMA_VERSION, + analyze_target_risks, +) +from tests.workflow_cross_field.support_graph import ( + SUPPORT_GRAPH_JSON_PATH, + SUPPORT_GRAPH_MARKDOWN_PATH, + render_support_graph_json, + render_support_graph_markdown, +) + + +ROOT = Path(__file__).parent +SCENARIOS_DIR = ROOT / "scenarios" +TARGETS_DIR = ROOT / "targets" +RESULTS_DIR = ROOT / "results" +SUMMARY_PATH = ROOT / "SUMMARY.md" +TARGET_RISKS_JSON_PATH = ROOT / "target_risks.json" +TARGET_RISKS_MARKDOWN_PATH = ROOT / "TARGET_RISKS.md" +SEEDS = (7, 29) +SCHEMA_VERSION = 1 +ADAPTERS: tuple[AdapterType, ...] = ( + "model", + "real_route", + "real_coordinator", + "browser_smoke", +) + + +def _json_bytes(payload: object) -> bytes: + return (json.dumps(payload, indent=2, sort_keys=True) + "\n").encode() + + +def render_artifacts() -> dict[Path, bytes]: + rendered: dict[Path, bytes] = {} + records: list[InteractionCoverage] = [ + *REAL_ADAPTER_COVERAGE, + *BROWSER_SMOKE_COVERAGE, + ] + + for scenario in CROSS_FIELD_SCENARIOS: + path = SCENARIOS_DIR / f"{scenario.scenario_id}.json" + rendered[path] = _json_bytes( + { + "schema_version": SCHEMA_VERSION, + "artifact_type": "scenario", + "scenario_id": scenario.scenario_id, + "adapter": scenario.adapter, + "fields": scenario.fields, + "invariants": scenario.invariants, + "actions": tuple(action.__name__ for action in scenario.actions), + "must_exercise": scenario.must_exercise, + "expected_result": scenario.expected_result, + } + ) + record = scenario.coverage_record("tests/workflow_scenarios/test_cross_field_analysis.py") + records.append(record) + result_path = RESULTS_DIR / f"{scenario.scenario_id}.json" + rendered[result_path] = _json_bytes(_result_payload(record, source="curated")) + + for target in DEFAULT_TARGETS: + result_ids = tuple(f"{target.target_id}_seed_{seed}" for seed in SEEDS) + rendered[TARGETS_DIR / f"{target.target_id}.json"] = _json_bytes( + { + "schema_version": SCHEMA_VERSION, + "artifact_type": "generation_target", + "target_id": target.target_id, + "fields": target.fields, + "invariants": target.invariants, + "must_exercise": target.must_exercise, + "max_steps": target.max_steps, + "result_ids": result_ids, + } + ) + + with tempfile.TemporaryDirectory(prefix="moto-workflow-artifacts-"): + # The executable model performs only in-memory/path-containment checks here. A stable + # logical root keeps planner signatures and generated artifacts process-independent. + runtime_root = Path("/moto-workflow-artifacts") + for target in DEFAULT_TARGETS: + for seed in SEEDS: + run = run_generated_scenario( + WorkflowModel(runtime_root=runtime_root / target.target_id / str(seed)), + target, + seed=seed, + ) + scenario_id = f"{target.target_id}_seed_{seed}" + path = RESULTS_DIR / f"{scenario_id}.json" + payload = { + "schema_version": SCHEMA_VERSION, + "artifact_type": "result", + "scenario_id": scenario_id, + "target_id": target.target_id, + "target_artifact": f"targets/{target.target_id}.json", + "adapter": "model", + "result": "passed", + "seed": seed, + "fields": run.fields, + "observed_fields": sorted(run.observed_fields), + "invariants": run.invariants, + "exercised_invariants": sorted(run.exercised_invariants), + "action_ids": run.action_ids, + "replay": run.replay, + "evidence": sorted(run.observed_evidence), + "diagnostics": { + "final_mode": run.final_mode.value, + "final_phase": run.final_phase.value, + }, + } + rendered[path] = _json_bytes(payload) + records.append( + InteractionCoverage( + scenario_id=scenario_id, + fields=run.fields, + invariants=run.invariants, + adapter="model", + result="passed", + test_file="tests/workflow_recombinatory/test_mode_aware_generator.py", + seed=seed, + replay=run.replay, + diagnostics=payload["diagnostics"], + evidence=tuple(sorted(run.observed_evidence)), + ) + ) + + for record in REAL_ADAPTER_COVERAGE: + rendered[RESULTS_DIR / f"{record.scenario_id}.json"] = _json_bytes( + _result_payload(record, source="real_adapter") + ) + for record in BROWSER_SMOKE_COVERAGE: + rendered[RESULTS_DIR / f"{record.scenario_id}.json"] = _json_bytes( + _result_payload(record, source="browser_smoke") + ) + + lines = [ + "# Cross-Field Coverage Summary", + "", + "Generated deterministically by `tests.workflow_cross_field.artifacts`.", + "", + f"Schema version: {SCHEMA_VERSION}.", + "", + "| Product law | model | real_route | real_coordinator | browser_smoke |", + "|---|---|---|---|---|", + ] + for invariant in sorted(INVARIANTS_BY_ID): + cells = [_status_cell(records, invariant, adapter) for adapter in ADAPTERS] + lines.append(f"| `{invariant}` | {' | '.join(cells)} |") + rendered[SUMMARY_PATH] = ("\n".join(lines) + "\n").encode() + analyses = analyze_target_risks(records) + rendered[TARGET_RISKS_JSON_PATH] = _json_bytes( + { + "schema_version": TARGET_RISK_SCHEMA_VERSION, + "artifact_type": "target_risk_analysis", + "risks": [_risk_payload(analysis) for analysis in analyses], + } + ) + risk_lines = [ + "# Inverse Target-Risk Analysis", + "", + "Generated deterministically by `tests.workflow_cross_field.artifacts` from Build A in-memory coverage records.", + "", + f"Schema version: {TARGET_RISK_SCHEMA_VERSION}.", + "", + "| Rank | Target risk | Priority | Gap score | Adapter support | Missing isolated artifacts |", + "|---:|---|---|---:|---|---|", + ] + for analysis in analyses: + support = "; ".join(f"{item.adapter}: {item.status}" for item in analysis.support) + missing = ", ".join(f"`{item}`" for item in analysis.missing_isolated_artifacts) or "none" + risk_lines.append( + f"| {analysis.rank} | `{analysis.risk.risk_id}` — {analysis.risk.title} | " + f"{analysis.risk.priority} | {analysis.gap_score} | {support} | {missing} |" + ) + rendered[TARGET_RISKS_MARKDOWN_PATH] = ("\n".join(risk_lines) + "\n").encode() + rendered[SUPPORT_GRAPH_JSON_PATH] = render_support_graph_json(records) + rendered[SUPPORT_GRAPH_MARKDOWN_PATH] = render_support_graph_markdown(records) + return rendered + + +def _risk_payload(analysis) -> dict[str, object]: + return { + "risk_id": analysis.risk.risk_id, + "title": analysis.risk.title, + "description": analysis.risk.description, + "priority": analysis.risk.priority, + "rank": analysis.rank, + "gap_score": analysis.gap_score, + "required_fields": analysis.risk.required_fields, + "supporting_invariants": analysis.risk.supporting_invariants, + "required_adapters": analysis.risk.required_adapters, + "existing_scenarios": analysis.existing_scenarios, + "missing_isolated_artifacts": analysis.missing_isolated_artifacts, + "support": [ + { + "adapter": item.adapter, + "status": item.status, + "result_statuses": item.result_statuses, + "scenario_ids": item.scenario_ids, + "missing_invariants": item.missing_invariants, + } + for item in analysis.support + ], + } + + +def _result_payload(record: InteractionCoverage, *, source: str) -> dict[str, object]: + return { + "schema_version": SCHEMA_VERSION, + "artifact_type": "result", + "source": source, + "scenario_id": record.scenario_id, + "adapter": record.adapter, + "result": record.result, + "fields": record.fields, + "invariants": record.invariants, + "test_file": record.test_file, + "seed": record.seed, + "replay": record.replay, + "diagnostics": record.diagnostics or {}, + "evidence": record.evidence, + } + + +def _status_cell( + records: list[InteractionCoverage], invariant: str, adapter: AdapterType +) -> str: + matches = [ + record for record in records + if invariant in record.invariants and record.adapter == adapter + ] + if not matches: + return "uncovered" + statuses = sorted({record.result for record in matches}) + reasons = sorted({ + str(record.diagnostics.get("reason")) + for record in matches + if record.result in {"blocked", "skipped"} + and record.diagnostics + and record.diagnostics.get("reason") + }) + return ", ".join(statuses) + (f" ({'; '.join(reasons)})" if reasons else "") + + +def write_artifacts(*, check: bool = False) -> int: + rendered = render_artifacts() + stale = [path for path, content in rendered.items() if not path.exists() or path.read_bytes() != content] + expected = set(rendered) + extras = ( + set(SCENARIOS_DIR.glob("*.json")) + | set(TARGETS_DIR.glob("*.json")) + | set(RESULTS_DIR.glob("*.json")) + ) + stale.extend(sorted(extras - expected)) + if check: + return 1 if stale else 0 + for path in extras - expected: + path.unlink() + for path, content in rendered.items(): + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(content) + return 0 + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("--check", action="store_true") + args = parser.parse_args() + raise SystemExit(write_artifacts(check=args.check)) diff --git a/tests/workflow_cross_field/results/browser_autonomous_start_stop_preserves_single_ui_owner.json b/tests/workflow_cross_field/results/browser_autonomous_start_stop_preserves_single_ui_owner.json new file mode 100644 index 0000000..bba51af --- /dev/null +++ b/tests/workflow_cross_field/results/browser_autonomous_start_stop_preserves_single_ui_owner.json @@ -0,0 +1,26 @@ +{ + "adapter": "browser_smoke", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "papers_only_start_payload", + "running_owner_controls", + "stop_restores_controls" + ], + "fields": [ + "runtime_exclusivity", + "allowed_outputs", + "websocket_api_contracts" + ], + "invariants": [ + "runtime.single_active_workflow", + "outputs.at_least_one_output_enabled" + ], + "replay": [], + "result": "passed", + "scenario_id": "browser_autonomous_start_stop_preserves_single_ui_owner", + "schema_version": 1, + "seed": null, + "source": "browser_smoke", + "test_file": "tests/workflow_browser_smoke/autonomous-controls.spec.js" +} diff --git a/tests/workflow_cross_field/results/browser_hosted_startup_respects_desktop_capability_boundaries.json b/tests/workflow_cross_field/results/browser_hosted_startup_respects_desktop_capability_boundaries.json new file mode 100644 index 0000000..45aa292 --- /dev/null +++ b/tests/workflow_cross_field/results/browser_hosted_startup_respects_desktop_capability_boundaries.json @@ -0,0 +1,24 @@ +{ + "adapter": "browser_smoke", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "hosted_openrouter_startup", + "desktop_provider_paths_not_requested" + ], + "fields": [ + "websocket_api_contracts", + "proof_runtime_gating", + "runtime_exclusivity" + ], + "invariants": [ + "api.hosted_desktop_only_routes_unavailable" + ], + "replay": [], + "result": "passed", + "scenario_id": "browser_hosted_startup_respects_desktop_capability_boundaries", + "schema_version": 1, + "seed": null, + "source": "browser_smoke", + "test_file": "tests/workflow_browser_smoke/hosted-capabilities.spec.js" +} diff --git a/tests/workflow_cross_field/results/browser_mode_switching_preserves_in_memory_default_and_developer_gate.json b/tests/workflow_cross_field/results/browser_mode_switching_preserves_in_memory_default_and_developer_gate.json new file mode 100644 index 0000000..a6f2f5e --- /dev/null +++ b/tests/workflow_cross_field/results/browser_mode_switching_preserves_in_memory_default_and_developer_gate.json @@ -0,0 +1,27 @@ +{ + "adapter": "browser_smoke", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "autonomous_default_mode", + "manual_mode_switch", + "leanoj_hidden_before_developer_gate", + "leanoj_visible_after_developer_gate", + "reload_restores_autonomous_mode" + ], + "fields": [ + "runtime_exclusivity", + "workflow_filesystem_state", + "websocket_api_contracts" + ], + "invariants": [ + "runtime.single_active_workflow" + ], + "replay": [], + "result": "passed", + "scenario_id": "browser_mode_switching_preserves_in_memory_default_and_developer_gate", + "schema_version": 1, + "seed": null, + "source": "browser_smoke", + "test_file": "tests/workflow_browser_smoke/mode-switching.spec.js" +} diff --git a/tests/workflow_cross_field/results/browser_overflow_activity_persists_with_attribution_and_terminal_deduplication.json b/tests/workflow_cross_field/results/browser_overflow_activity_persists_with_attribution_and_terminal_deduplication.json new file mode 100644 index 0000000..f7d002b --- /dev/null +++ b/tests/workflow_cross_field/results/browser_overflow_activity_persists_with_attribution_and_terminal_deduplication.json @@ -0,0 +1,29 @@ +{ + "adapter": "browser_smoke", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "configured_and_effective_route_visible", + "proof_overflow_keeps_parent_active", + "fatal_stop_not_duplicated", + "activity_restored_after_reload" + ], + "fields": [ + "websocket_api_contracts", + "workflow_filesystem_state", + "provider_pause_resume" + ], + "invariants": [ + "events.context_overflow_route_identity", + "events.context_overflow_persists_across_reload", + "events.context_overflow_terminal_stop_once", + "events.proof_context_overflow_nonfatal" + ], + "replay": [], + "result": "passed", + "scenario_id": "browser_overflow_activity_persists_with_attribution_and_terminal_deduplication", + "schema_version": 1, + "seed": null, + "source": "browser_smoke", + "test_file": "tests/workflow_browser_smoke/overflow-activity.spec.js" +} diff --git a/tests/workflow_cross_field/results/generated_allowed_outputs_provider_pause_resume_seed_29.json b/tests/workflow_cross_field/results/generated_allowed_outputs_provider_pause_resume_seed_29.json new file mode 100644 index 0000000..e197d1b --- /dev/null +++ b/tests/workflow_cross_field/results/generated_allowed_outputs_provider_pause_resume_seed_29.json @@ -0,0 +1,61 @@ +{ + "action_ids": [ + "start_autonomous_proofs_only", + "complete_topic_exploration", + "simulate_credit_exhaustion", + "complete_brainstorm", + "stop", + "resume", + "reset_credit" + ], + "adapter": "model", + "artifact_type": "result", + "diagnostics": { + "final_mode": "autonomous", + "final_phase": "tier1_aggregation" + }, + "evidence": [ + "provider_pause", + "reset_credit", + "stop_resume" + ], + "exercised_invariants": [ + "outputs.proofs_only_no_paper_phase", + "provider.pause_preserves_checkpoint", + "provider.reset_wakes_without_corrupting_checkpoint", + "provider.stop_resume_preserves_pause" + ], + "fields": [ + "allowed_outputs", + "provider_pause_resume", + "workflow_filesystem_state" + ], + "invariants": [ + "outputs.proofs_only_no_paper_phase", + "provider.pause_preserves_checkpoint", + "provider.stop_resume_preserves_pause", + "provider.reset_wakes_without_corrupting_checkpoint" + ], + "observed_fields": [ + "allowed_outputs", + "proof_runtime_gating", + "provider_pause_resume", + "runtime_exclusivity", + "workflow_filesystem_state" + ], + "replay": [ + "start_autonomous(lean_enabled=True, papers=False, proofs=True)", + "complete_topic_exploration()", + "simulate_credit_exhaustion()", + "complete_brainstorm()", + "stop()", + "resume()", + "reset_credit()" + ], + "result": "passed", + "scenario_id": "generated_allowed_outputs_provider_pause_resume_seed_29", + "schema_version": 1, + "seed": 29, + "target_artifact": "targets/generated_allowed_outputs_provider_pause_resume.json", + "target_id": "generated_allowed_outputs_provider_pause_resume" +} diff --git a/tests/workflow_cross_field/results/generated_allowed_outputs_provider_pause_resume_seed_7.json b/tests/workflow_cross_field/results/generated_allowed_outputs_provider_pause_resume_seed_7.json new file mode 100644 index 0000000..a4ba91b --- /dev/null +++ b/tests/workflow_cross_field/results/generated_allowed_outputs_provider_pause_resume_seed_7.json @@ -0,0 +1,61 @@ +{ + "action_ids": [ + "start_autonomous_proofs_only", + "complete_topic_exploration", + "simulate_credit_exhaustion", + "complete_brainstorm", + "stop", + "resume", + "reset_credit" + ], + "adapter": "model", + "artifact_type": "result", + "diagnostics": { + "final_mode": "autonomous", + "final_phase": "tier1_aggregation" + }, + "evidence": [ + "provider_pause", + "reset_credit", + "stop_resume" + ], + "exercised_invariants": [ + "outputs.proofs_only_no_paper_phase", + "provider.pause_preserves_checkpoint", + "provider.reset_wakes_without_corrupting_checkpoint", + "provider.stop_resume_preserves_pause" + ], + "fields": [ + "allowed_outputs", + "provider_pause_resume", + "workflow_filesystem_state" + ], + "invariants": [ + "outputs.proofs_only_no_paper_phase", + "provider.pause_preserves_checkpoint", + "provider.stop_resume_preserves_pause", + "provider.reset_wakes_without_corrupting_checkpoint" + ], + "observed_fields": [ + "allowed_outputs", + "proof_runtime_gating", + "provider_pause_resume", + "runtime_exclusivity", + "workflow_filesystem_state" + ], + "replay": [ + "start_autonomous(lean_enabled=True, papers=False, proofs=True)", + "complete_topic_exploration()", + "simulate_credit_exhaustion()", + "complete_brainstorm()", + "stop()", + "resume()", + "reset_credit()" + ], + "result": "passed", + "scenario_id": "generated_allowed_outputs_provider_pause_resume_seed_7", + "schema_version": 1, + "seed": 7, + "target_artifact": "targets/generated_allowed_outputs_provider_pause_resume.json", + "target_id": "generated_allowed_outputs_provider_pause_resume" +} diff --git a/tests/workflow_cross_field/results/generated_assistant_prompt_context_seed_29.json b/tests/workflow_cross_field/results/generated_assistant_prompt_context_seed_29.json new file mode 100644 index 0000000..430279d --- /dev/null +++ b/tests/workflow_cross_field/results/generated_assistant_prompt_context_seed_29.json @@ -0,0 +1,52 @@ +{ + "action_ids": [ + "start_autonomous_papers_and_proofs", + "refresh_assistant_pack", + "prepare_prompt_context", + "disable_session_history" + ], + "adapter": "model", + "artifact_type": "result", + "diagnostics": { + "final_mode": "autonomous", + "final_phase": "topic_exploration" + }, + "evidence": [ + "assistant_disable", + "assistant_pack", + "prompt_context" + ], + "exercised_invariants": [ + "assistant.disable_clears_live_pack", + "assistant.no_validator_injection", + "prompt.validator_excludes_assistant_memory" + ], + "fields": [ + "assistant_memory", + "prompt_context" + ], + "invariants": [ + "assistant.disable_clears_live_pack", + "assistant.no_validator_injection", + "prompt.validator_excludes_assistant_memory" + ], + "observed_fields": [ + "allowed_outputs", + "assistant_memory", + "prompt_context", + "proof_scope_isolation", + "websocket_api_contracts" + ], + "replay": [ + "start_autonomous(lean_enabled=True, papers=True, proofs=True)", + "refresh_assistant_pack(proof_ids=('prior-proof-1', 'prior-proof-2'))", + "prepare_prompt_context()", + "toggle_session_history_memory(enabled=False)" + ], + "result": "passed", + "scenario_id": "generated_assistant_prompt_context_seed_29", + "schema_version": 1, + "seed": 29, + "target_artifact": "targets/generated_assistant_prompt_context.json", + "target_id": "generated_assistant_prompt_context" +} diff --git a/tests/workflow_cross_field/results/generated_assistant_prompt_context_seed_7.json b/tests/workflow_cross_field/results/generated_assistant_prompt_context_seed_7.json new file mode 100644 index 0000000..2bf173d --- /dev/null +++ b/tests/workflow_cross_field/results/generated_assistant_prompt_context_seed_7.json @@ -0,0 +1,52 @@ +{ + "action_ids": [ + "start_autonomous_papers_and_proofs", + "refresh_assistant_pack", + "prepare_prompt_context", + "disable_session_history" + ], + "adapter": "model", + "artifact_type": "result", + "diagnostics": { + "final_mode": "autonomous", + "final_phase": "topic_exploration" + }, + "evidence": [ + "assistant_disable", + "assistant_pack", + "prompt_context" + ], + "exercised_invariants": [ + "assistant.disable_clears_live_pack", + "assistant.no_validator_injection", + "prompt.validator_excludes_assistant_memory" + ], + "fields": [ + "assistant_memory", + "prompt_context" + ], + "invariants": [ + "assistant.disable_clears_live_pack", + "assistant.no_validator_injection", + "prompt.validator_excludes_assistant_memory" + ], + "observed_fields": [ + "allowed_outputs", + "assistant_memory", + "prompt_context", + "proof_scope_isolation", + "websocket_api_contracts" + ], + "replay": [ + "start_autonomous(lean_enabled=True, papers=True, proofs=True)", + "refresh_assistant_pack(proof_ids=('prior-proof-1', 'prior-proof-2'))", + "prepare_prompt_context()", + "toggle_session_history_memory(enabled=False)" + ], + "result": "passed", + "scenario_id": "generated_assistant_prompt_context_seed_7", + "schema_version": 1, + "seed": 7, + "target_artifact": "targets/generated_assistant_prompt_context.json", + "target_id": "generated_assistant_prompt_context" +} diff --git a/tests/workflow_cross_field/results/generated_autonomous_paper_checkpoint_seed_29.json b/tests/workflow_cross_field/results/generated_autonomous_paper_checkpoint_seed_29.json new file mode 100644 index 0000000..b65df29 --- /dev/null +++ b/tests/workflow_cross_field/results/generated_autonomous_paper_checkpoint_seed_29.json @@ -0,0 +1,57 @@ +{ + "action_ids": [ + "start_autonomous_papers_and_proofs", + "attempt_conflicting_start", + "complete_topic_exploration", + "force_paper_writing", + "enter_autonomous_paper_checkpoint", + "complete_autonomous_paper_checkpoint" + ], + "adapter": "model", + "artifact_type": "result", + "diagnostics": { + "final_mode": "autonomous", + "final_phase": "topic_exploration" + }, + "evidence": [ + "paper_checkpoint", + "scoped_event", + "start_blocked" + ], + "exercised_invariants": [ + "runtime.single_active_workflow" + ], + "fields": [ + "runtime_exclusivity", + "workflow_filesystem_state", + "websocket_api_contracts" + ], + "invariants": [ + "runtime.single_active_workflow" + ], + "observed_fields": [ + "allowed_outputs", + "assistant_memory", + "prompt_context", + "proof_runtime_gating", + "provider_pause_resume", + "runtime_exclusivity", + "websocket_api_contracts", + "workflow_filesystem_state" + ], + "replay": [ + "start_autonomous(lean_enabled=True, papers=True, proofs=True)", + "start_manual_compiler()", + "complete_topic_exploration()", + "force_paper_writing()", + "complete_brainstorm()", + "enter_autonomous_paper_checkpoint()", + "complete_autonomous_paper_checkpoint()" + ], + "result": "passed", + "scenario_id": "generated_autonomous_paper_checkpoint_seed_29", + "schema_version": 1, + "seed": 29, + "target_artifact": "targets/generated_autonomous_paper_checkpoint.json", + "target_id": "generated_autonomous_paper_checkpoint" +} diff --git a/tests/workflow_cross_field/results/generated_autonomous_paper_checkpoint_seed_7.json b/tests/workflow_cross_field/results/generated_autonomous_paper_checkpoint_seed_7.json new file mode 100644 index 0000000..f113571 --- /dev/null +++ b/tests/workflow_cross_field/results/generated_autonomous_paper_checkpoint_seed_7.json @@ -0,0 +1,57 @@ +{ + "action_ids": [ + "start_autonomous_papers_and_proofs", + "attempt_conflicting_start", + "complete_topic_exploration", + "force_paper_writing", + "enter_autonomous_paper_checkpoint", + "complete_autonomous_paper_checkpoint" + ], + "adapter": "model", + "artifact_type": "result", + "diagnostics": { + "final_mode": "autonomous", + "final_phase": "topic_exploration" + }, + "evidence": [ + "paper_checkpoint", + "scoped_event", + "start_blocked" + ], + "exercised_invariants": [ + "runtime.single_active_workflow" + ], + "fields": [ + "runtime_exclusivity", + "workflow_filesystem_state", + "websocket_api_contracts" + ], + "invariants": [ + "runtime.single_active_workflow" + ], + "observed_fields": [ + "allowed_outputs", + "assistant_memory", + "prompt_context", + "proof_runtime_gating", + "provider_pause_resume", + "runtime_exclusivity", + "websocket_api_contracts", + "workflow_filesystem_state" + ], + "replay": [ + "start_autonomous(lean_enabled=True, papers=True, proofs=True)", + "start_manual_compiler()", + "complete_topic_exploration()", + "force_paper_writing()", + "complete_brainstorm()", + "enter_autonomous_paper_checkpoint()", + "complete_autonomous_paper_checkpoint()" + ], + "result": "passed", + "scenario_id": "generated_autonomous_paper_checkpoint_seed_7", + "schema_version": 1, + "seed": 7, + "target_artifact": "targets/generated_autonomous_paper_checkpoint.json", + "target_id": "generated_autonomous_paper_checkpoint" +} diff --git a/tests/workflow_cross_field/results/generated_context_overflow_contract_seed_29.json b/tests/workflow_cross_field/results/generated_context_overflow_contract_seed_29.json new file mode 100644 index 0000000..2c2fad3 --- /dev/null +++ b/tests/workflow_cross_field/results/generated_context_overflow_contract_seed_29.json @@ -0,0 +1,51 @@ +{ + "action_ids": [ + "start_manual_aggregator", + "emit_context_overflow_contract_events" + ], + "adapter": "model", + "artifact_type": "result", + "diagnostics": { + "final_mode": "manual_aggregator", + "final_phase": "manual_aggregation" + }, + "evidence": [ + "context_overflow" + ], + "exercised_invariants": [ + "events.context_overflow_persists_across_reload", + "events.context_overflow_route_identity", + "events.context_overflow_terminal_stop_once", + "events.proof_context_overflow_nonfatal" + ], + "fields": [ + "websocket_api_contracts", + "workflow_filesystem_state", + "provider_pause_resume", + "runtime_exclusivity", + "proof_runtime_gating" + ], + "invariants": [ + "events.context_overflow_route_identity", + "events.context_overflow_persists_across_reload", + "events.context_overflow_terminal_stop_once", + "events.proof_context_overflow_nonfatal" + ], + "observed_fields": [ + "proof_runtime_gating", + "provider_pause_resume", + "runtime_exclusivity", + "websocket_api_contracts", + "workflow_filesystem_state" + ], + "replay": [ + "start_manual_aggregator()", + "emit_context_overflow_contract_events()" + ], + "result": "passed", + "scenario_id": "generated_context_overflow_contract_seed_29", + "schema_version": 1, + "seed": 29, + "target_artifact": "targets/generated_context_overflow_contract.json", + "target_id": "generated_context_overflow_contract" +} diff --git a/tests/workflow_cross_field/results/generated_context_overflow_contract_seed_7.json b/tests/workflow_cross_field/results/generated_context_overflow_contract_seed_7.json new file mode 100644 index 0000000..33198a0 --- /dev/null +++ b/tests/workflow_cross_field/results/generated_context_overflow_contract_seed_7.json @@ -0,0 +1,51 @@ +{ + "action_ids": [ + "start_manual_aggregator", + "emit_context_overflow_contract_events" + ], + "adapter": "model", + "artifact_type": "result", + "diagnostics": { + "final_mode": "manual_aggregator", + "final_phase": "manual_aggregation" + }, + "evidence": [ + "context_overflow" + ], + "exercised_invariants": [ + "events.context_overflow_persists_across_reload", + "events.context_overflow_route_identity", + "events.context_overflow_terminal_stop_once", + "events.proof_context_overflow_nonfatal" + ], + "fields": [ + "websocket_api_contracts", + "workflow_filesystem_state", + "provider_pause_resume", + "runtime_exclusivity", + "proof_runtime_gating" + ], + "invariants": [ + "events.context_overflow_route_identity", + "events.context_overflow_persists_across_reload", + "events.context_overflow_terminal_stop_once", + "events.proof_context_overflow_nonfatal" + ], + "observed_fields": [ + "proof_runtime_gating", + "provider_pause_resume", + "runtime_exclusivity", + "websocket_api_contracts", + "workflow_filesystem_state" + ], + "replay": [ + "start_manual_aggregator()", + "emit_context_overflow_contract_events()" + ], + "result": "passed", + "scenario_id": "generated_context_overflow_contract_seed_7", + "schema_version": 1, + "seed": 7, + "target_artifact": "targets/generated_context_overflow_contract.json", + "target_id": "generated_context_overflow_contract" +} diff --git a/tests/workflow_cross_field/results/generated_leanoj_durable_lifecycle_seed_29.json b/tests/workflow_cross_field/results/generated_leanoj_durable_lifecycle_seed_29.json new file mode 100644 index 0000000..dcaa4e1 --- /dev/null +++ b/tests/workflow_cross_field/results/generated_leanoj_durable_lifecycle_seed_29.json @@ -0,0 +1,61 @@ +{ + "action_ids": [ + "start_leanoj", + "skip_leanoj_brainstorm", + "force_leanoj_brainstorm", + "stop", + "resume", + "attempt_conflicting_start", + "edit_leanoj_master_proof", + "clear_leanoj_state" + ], + "adapter": "model", + "artifact_type": "result", + "diagnostics": { + "final_mode": "none", + "final_phase": "idle" + }, + "evidence": [ + "leanoj_clear", + "leanoj_draft", + "leanoj_force", + "leanoj_skip", + "start_blocked", + "stop_resume" + ], + "exercised_invariants": [ + "provider.stop_resume_preserves_pause", + "runtime.single_active_workflow" + ], + "fields": [ + "runtime_exclusivity", + "workflow_filesystem_state" + ], + "invariants": [ + "runtime.single_active_workflow", + "provider.stop_resume_preserves_pause" + ], + "observed_fields": [ + "prompt_context", + "provider_pause_resume", + "runtime_exclusivity", + "websocket_api_contracts", + "workflow_filesystem_state" + ], + "replay": [ + "start_leanoj()", + "skip_leanoj_brainstorm()", + "force_leanoj_brainstorm()", + "stop()", + "resume()", + "start_manual_compiler()", + "edit_leanoj_master_proof()", + "clear()" + ], + "result": "passed", + "scenario_id": "generated_leanoj_durable_lifecycle_seed_29", + "schema_version": 1, + "seed": 29, + "target_artifact": "targets/generated_leanoj_durable_lifecycle.json", + "target_id": "generated_leanoj_durable_lifecycle" +} diff --git a/tests/workflow_cross_field/results/generated_leanoj_durable_lifecycle_seed_7.json b/tests/workflow_cross_field/results/generated_leanoj_durable_lifecycle_seed_7.json new file mode 100644 index 0000000..f64315f --- /dev/null +++ b/tests/workflow_cross_field/results/generated_leanoj_durable_lifecycle_seed_7.json @@ -0,0 +1,61 @@ +{ + "action_ids": [ + "start_leanoj", + "skip_leanoj_brainstorm", + "stop", + "resume", + "force_leanoj_brainstorm", + "attempt_conflicting_start", + "edit_leanoj_master_proof", + "clear_leanoj_state" + ], + "adapter": "model", + "artifact_type": "result", + "diagnostics": { + "final_mode": "none", + "final_phase": "idle" + }, + "evidence": [ + "leanoj_clear", + "leanoj_draft", + "leanoj_force", + "leanoj_skip", + "start_blocked", + "stop_resume" + ], + "exercised_invariants": [ + "provider.stop_resume_preserves_pause", + "runtime.single_active_workflow" + ], + "fields": [ + "runtime_exclusivity", + "workflow_filesystem_state" + ], + "invariants": [ + "runtime.single_active_workflow", + "provider.stop_resume_preserves_pause" + ], + "observed_fields": [ + "prompt_context", + "provider_pause_resume", + "runtime_exclusivity", + "websocket_api_contracts", + "workflow_filesystem_state" + ], + "replay": [ + "start_leanoj()", + "skip_leanoj_brainstorm()", + "stop()", + "resume()", + "force_leanoj_brainstorm()", + "start_manual_compiler()", + "edit_leanoj_master_proof()", + "clear()" + ], + "result": "passed", + "scenario_id": "generated_leanoj_durable_lifecycle_seed_7", + "schema_version": 1, + "seed": 7, + "target_artifact": "targets/generated_leanoj_durable_lifecycle.json", + "target_id": "generated_leanoj_durable_lifecycle" +} diff --git a/tests/workflow_cross_field/results/generated_manual_aggregator_lifecycle_seed_29.json b/tests/workflow_cross_field/results/generated_manual_aggregator_lifecycle_seed_29.json new file mode 100644 index 0000000..afd59ea --- /dev/null +++ b/tests/workflow_cross_field/results/generated_manual_aggregator_lifecycle_seed_29.json @@ -0,0 +1,47 @@ +{ + "action_ids": [ + "start_manual_aggregator", + "attempt_conflicting_start", + "accept_manual_aggregator_submission", + "clear_manual_aggregator_state" + ], + "adapter": "model", + "artifact_type": "result", + "diagnostics": { + "final_mode": "none", + "final_phase": "idle" + }, + "evidence": [ + "manual_aggregator_accept", + "manual_aggregator_clear", + "start_blocked" + ], + "exercised_invariants": [ + "runtime.single_active_workflow" + ], + "fields": [ + "runtime_exclusivity", + "workflow_filesystem_state" + ], + "invariants": [ + "runtime.single_active_workflow" + ], + "observed_fields": [ + "prompt_context", + "runtime_exclusivity", + "websocket_api_contracts", + "workflow_filesystem_state" + ], + "replay": [ + "start_manual_aggregator()", + "start_manual_compiler()", + "accept_manual_aggregator_submission()", + "clear()" + ], + "result": "passed", + "scenario_id": "generated_manual_aggregator_lifecycle_seed_29", + "schema_version": 1, + "seed": 29, + "target_artifact": "targets/generated_manual_aggregator_lifecycle.json", + "target_id": "generated_manual_aggregator_lifecycle" +} diff --git a/tests/workflow_cross_field/results/generated_manual_aggregator_lifecycle_seed_7.json b/tests/workflow_cross_field/results/generated_manual_aggregator_lifecycle_seed_7.json new file mode 100644 index 0000000..a0b38dc --- /dev/null +++ b/tests/workflow_cross_field/results/generated_manual_aggregator_lifecycle_seed_7.json @@ -0,0 +1,47 @@ +{ + "action_ids": [ + "start_manual_aggregator", + "attempt_conflicting_start", + "accept_manual_aggregator_submission", + "clear_manual_aggregator_state" + ], + "adapter": "model", + "artifact_type": "result", + "diagnostics": { + "final_mode": "none", + "final_phase": "idle" + }, + "evidence": [ + "manual_aggregator_accept", + "manual_aggregator_clear", + "start_blocked" + ], + "exercised_invariants": [ + "runtime.single_active_workflow" + ], + "fields": [ + "runtime_exclusivity", + "workflow_filesystem_state" + ], + "invariants": [ + "runtime.single_active_workflow" + ], + "observed_fields": [ + "prompt_context", + "runtime_exclusivity", + "websocket_api_contracts", + "workflow_filesystem_state" + ], + "replay": [ + "start_manual_aggregator()", + "start_manual_compiler()", + "accept_manual_aggregator_submission()", + "clear()" + ], + "result": "passed", + "scenario_id": "generated_manual_aggregator_lifecycle_seed_7", + "schema_version": 1, + "seed": 7, + "target_artifact": "targets/generated_manual_aggregator_lifecycle.json", + "target_id": "generated_manual_aggregator_lifecycle" +} diff --git a/tests/workflow_cross_field/results/generated_manual_scope_clear_archive_seed_29.json b/tests/workflow_cross_field/results/generated_manual_scope_clear_archive_seed_29.json new file mode 100644 index 0000000..45ccf97 --- /dev/null +++ b/tests/workflow_cross_field/results/generated_manual_scope_clear_archive_seed_29.json @@ -0,0 +1,56 @@ +{ + "action_ids": [ + "start_manual_compiler", + "refresh_assistant_pack", + "enable_manual_lean", + "run_manual_proof_check", + "clear_manual_state" + ], + "adapter": "model", + "artifact_type": "result", + "diagnostics": { + "final_mode": "none", + "final_phase": "idle" + }, + "evidence": [ + "assistant_pack", + "manual_archive", + "manual_proof", + "scoped_event" + ], + "exercised_invariants": [ + "proof_scope.manual_clear_archives_active", + "proof_scope.manual_not_in_autonomous_current", + "state.clear_removes_active_preserves_history" + ], + "fields": [ + "proof_scope_isolation", + "workflow_filesystem_state", + "assistant_memory" + ], + "invariants": [ + "proof_scope.manual_not_in_autonomous_current", + "proof_scope.manual_clear_archives_active", + "state.clear_removes_active_preserves_history" + ], + "observed_fields": [ + "assistant_memory", + "prompt_context", + "proof_runtime_gating", + "proof_scope_isolation", + "workflow_filesystem_state" + ], + "replay": [ + "start_manual_compiler()", + "refresh_assistant_pack(proof_ids=('prior-proof-1', 'prior-proof-2'))", + "enable_manual_lean()", + "run_manual_proof_check()", + "clear()" + ], + "result": "passed", + "scenario_id": "generated_manual_scope_clear_archive_seed_29", + "schema_version": 1, + "seed": 29, + "target_artifact": "targets/generated_manual_scope_clear_archive.json", + "target_id": "generated_manual_scope_clear_archive" +} diff --git a/tests/workflow_cross_field/results/generated_manual_scope_clear_archive_seed_7.json b/tests/workflow_cross_field/results/generated_manual_scope_clear_archive_seed_7.json new file mode 100644 index 0000000..54a6358 --- /dev/null +++ b/tests/workflow_cross_field/results/generated_manual_scope_clear_archive_seed_7.json @@ -0,0 +1,56 @@ +{ + "action_ids": [ + "start_manual_compiler", + "refresh_assistant_pack", + "enable_manual_lean", + "run_manual_proof_check", + "clear_manual_state" + ], + "adapter": "model", + "artifact_type": "result", + "diagnostics": { + "final_mode": "none", + "final_phase": "idle" + }, + "evidence": [ + "assistant_pack", + "manual_archive", + "manual_proof", + "scoped_event" + ], + "exercised_invariants": [ + "proof_scope.manual_clear_archives_active", + "proof_scope.manual_not_in_autonomous_current", + "state.clear_removes_active_preserves_history" + ], + "fields": [ + "proof_scope_isolation", + "workflow_filesystem_state", + "assistant_memory" + ], + "invariants": [ + "proof_scope.manual_not_in_autonomous_current", + "proof_scope.manual_clear_archives_active", + "state.clear_removes_active_preserves_history" + ], + "observed_fields": [ + "assistant_memory", + "prompt_context", + "proof_runtime_gating", + "proof_scope_isolation", + "workflow_filesystem_state" + ], + "replay": [ + "start_manual_compiler()", + "refresh_assistant_pack(proof_ids=('prior-proof-1', 'prior-proof-2'))", + "enable_manual_lean()", + "run_manual_proof_check()", + "clear()" + ], + "result": "passed", + "scenario_id": "generated_manual_scope_clear_archive_seed_7", + "schema_version": 1, + "seed": 7, + "target_artifact": "targets/generated_manual_scope_clear_archive.json", + "target_id": "generated_manual_scope_clear_archive" +} diff --git a/tests/workflow_cross_field/results/generated_output_and_proof_runtime_gating_seed_29.json b/tests/workflow_cross_field/results/generated_output_and_proof_runtime_gating_seed_29.json new file mode 100644 index 0000000..3d11489 --- /dev/null +++ b/tests/workflow_cross_field/results/generated_output_and_proof_runtime_gating_seed_29.json @@ -0,0 +1,53 @@ +{ + "action_ids": [ + "start_autonomous_no_outputs", + "start_autonomous_papers_only", + "complete_topic_exploration", + "complete_brainstorm" + ], + "adapter": "model", + "artifact_type": "result", + "diagnostics": { + "final_mode": "autonomous", + "final_phase": "paper_title_selection" + }, + "evidence": [ + "no_outputs_rejected", + "papers_only" + ], + "exercised_invariants": [ + "outputs.at_least_one_output_enabled", + "outputs.papers_only_skips_proof_work", + "proof_runtime.no_lean_when_disabled", + "proof_runtime.no_smt_when_disabled" + ], + "fields": [ + "proof_runtime_gating", + "allowed_outputs" + ], + "invariants": [ + "outputs.at_least_one_output_enabled", + "outputs.papers_only_skips_proof_work", + "proof_runtime.no_lean_when_disabled", + "proof_runtime.no_smt_when_disabled" + ], + "observed_fields": [ + "allowed_outputs", + "proof_runtime_gating", + "provider_pause_resume", + "workflow_filesystem_state" + ], + "replay": [ + "start_autonomous_with_no_outputs()", + "start_autonomous(lean_enabled=False, papers=False, proofs=False)", + "start_autonomous(lean_enabled=False, papers=True, proofs=False)", + "complete_topic_exploration()", + "complete_brainstorm()" + ], + "result": "passed", + "scenario_id": "generated_output_and_proof_runtime_gating_seed_29", + "schema_version": 1, + "seed": 29, + "target_artifact": "targets/generated_output_and_proof_runtime_gating.json", + "target_id": "generated_output_and_proof_runtime_gating" +} diff --git a/tests/workflow_cross_field/results/generated_output_and_proof_runtime_gating_seed_7.json b/tests/workflow_cross_field/results/generated_output_and_proof_runtime_gating_seed_7.json new file mode 100644 index 0000000..95cf69e --- /dev/null +++ b/tests/workflow_cross_field/results/generated_output_and_proof_runtime_gating_seed_7.json @@ -0,0 +1,53 @@ +{ + "action_ids": [ + "start_autonomous_no_outputs", + "start_autonomous_papers_only", + "complete_topic_exploration", + "complete_brainstorm" + ], + "adapter": "model", + "artifact_type": "result", + "diagnostics": { + "final_mode": "autonomous", + "final_phase": "paper_title_selection" + }, + "evidence": [ + "no_outputs_rejected", + "papers_only" + ], + "exercised_invariants": [ + "outputs.at_least_one_output_enabled", + "outputs.papers_only_skips_proof_work", + "proof_runtime.no_lean_when_disabled", + "proof_runtime.no_smt_when_disabled" + ], + "fields": [ + "proof_runtime_gating", + "allowed_outputs" + ], + "invariants": [ + "outputs.at_least_one_output_enabled", + "outputs.papers_only_skips_proof_work", + "proof_runtime.no_lean_when_disabled", + "proof_runtime.no_smt_when_disabled" + ], + "observed_fields": [ + "allowed_outputs", + "proof_runtime_gating", + "provider_pause_resume", + "workflow_filesystem_state" + ], + "replay": [ + "start_autonomous_with_no_outputs()", + "start_autonomous(lean_enabled=False, papers=False, proofs=False)", + "start_autonomous(lean_enabled=False, papers=True, proofs=False)", + "complete_topic_exploration()", + "complete_brainstorm()" + ], + "result": "passed", + "scenario_id": "generated_output_and_proof_runtime_gating_seed_7", + "schema_version": 1, + "seed": 7, + "target_artifact": "targets/generated_output_and_proof_runtime_gating.json", + "target_id": "generated_output_and_proof_runtime_gating" +} diff --git a/tests/workflow_cross_field/results/generated_rag_prompt_boundaries_seed_29.json b/tests/workflow_cross_field/results/generated_rag_prompt_boundaries_seed_29.json new file mode 100644 index 0000000..40f7be0 --- /dev/null +++ b/tests/workflow_cross_field/results/generated_rag_prompt_boundaries_seed_29.json @@ -0,0 +1,60 @@ +{ + "action_ids": [ + "start_autonomous_papers_and_proofs", + "reject_mandatory_source_overflow", + "verify_rag_source_exclusion", + "prepare_prompt_context" + ], + "adapter": "model", + "artifact_type": "result", + "diagnostics": { + "final_mode": "autonomous", + "final_phase": "topic_exploration" + }, + "evidence": [ + "mandatory_source_overflow", + "prompt_context", + "rag_source_exclusion" + ], + "exercised_invariants": [ + "prompt.direct_sources_excluded_from_rag", + "prompt.generated_appendices_stripped", + "prompt.mandatory_source_overflow_fails_visible", + "prompt.proof_source_context_required", + "prompt.user_prompt_direct_injected" + ], + "fields": [ + "prompt_context", + "workflow_filesystem_state", + "proof_runtime_gating", + "websocket_api_contracts" + ], + "invariants": [ + "prompt.user_prompt_direct_injected", + "prompt.proof_source_context_required", + "prompt.direct_sources_excluded_from_rag", + "prompt.mandatory_source_overflow_fails_visible", + "prompt.generated_appendices_stripped" + ], + "observed_fields": [ + "allowed_outputs", + "assistant_memory", + "prompt_context", + "proof_runtime_gating", + "proof_scope_isolation", + "websocket_api_contracts", + "workflow_filesystem_state" + ], + "replay": [ + "start_autonomous(lean_enabled=True, papers=True, proofs=True)", + "reject_mandatory_source_overflow()", + "verify_rag_source_exclusion()", + "prepare_prompt_context()" + ], + "result": "passed", + "scenario_id": "generated_rag_prompt_boundaries_seed_29", + "schema_version": 1, + "seed": 29, + "target_artifact": "targets/generated_rag_prompt_boundaries.json", + "target_id": "generated_rag_prompt_boundaries" +} diff --git a/tests/workflow_cross_field/results/generated_rag_prompt_boundaries_seed_7.json b/tests/workflow_cross_field/results/generated_rag_prompt_boundaries_seed_7.json new file mode 100644 index 0000000..6747f23 --- /dev/null +++ b/tests/workflow_cross_field/results/generated_rag_prompt_boundaries_seed_7.json @@ -0,0 +1,60 @@ +{ + "action_ids": [ + "start_autonomous_papers_and_proofs", + "reject_mandatory_source_overflow", + "verify_rag_source_exclusion", + "prepare_prompt_context" + ], + "adapter": "model", + "artifact_type": "result", + "diagnostics": { + "final_mode": "autonomous", + "final_phase": "topic_exploration" + }, + "evidence": [ + "mandatory_source_overflow", + "prompt_context", + "rag_source_exclusion" + ], + "exercised_invariants": [ + "prompt.direct_sources_excluded_from_rag", + "prompt.generated_appendices_stripped", + "prompt.mandatory_source_overflow_fails_visible", + "prompt.proof_source_context_required", + "prompt.user_prompt_direct_injected" + ], + "fields": [ + "prompt_context", + "workflow_filesystem_state", + "proof_runtime_gating", + "websocket_api_contracts" + ], + "invariants": [ + "prompt.user_prompt_direct_injected", + "prompt.proof_source_context_required", + "prompt.direct_sources_excluded_from_rag", + "prompt.mandatory_source_overflow_fails_visible", + "prompt.generated_appendices_stripped" + ], + "observed_fields": [ + "allowed_outputs", + "assistant_memory", + "prompt_context", + "proof_runtime_gating", + "proof_scope_isolation", + "websocket_api_contracts", + "workflow_filesystem_state" + ], + "replay": [ + "start_autonomous(lean_enabled=True, papers=True, proofs=True)", + "reject_mandatory_source_overflow()", + "verify_rag_source_exclusion()", + "prepare_prompt_context()" + ], + "result": "passed", + "scenario_id": "generated_rag_prompt_boundaries_seed_7", + "schema_version": 1, + "seed": 7, + "target_artifact": "targets/generated_rag_prompt_boundaries.json", + "target_id": "generated_rag_prompt_boundaries" +} diff --git a/tests/workflow_cross_field/results/generated_runtime_start_exclusivity_seed_29.json b/tests/workflow_cross_field/results/generated_runtime_start_exclusivity_seed_29.json new file mode 100644 index 0000000..9293638 --- /dev/null +++ b/tests/workflow_cross_field/results/generated_runtime_start_exclusivity_seed_29.json @@ -0,0 +1,42 @@ +{ + "action_ids": [ + "start_autonomous_papers_and_proofs", + "attempt_conflicting_start" + ], + "adapter": "model", + "artifact_type": "result", + "diagnostics": { + "final_mode": "autonomous", + "final_phase": "topic_exploration" + }, + "evidence": [ + "start_blocked" + ], + "exercised_invariants": [ + "runtime.single_active_workflow" + ], + "fields": [ + "runtime_exclusivity", + "websocket_api_contracts" + ], + "invariants": [ + "runtime.single_active_workflow" + ], + "observed_fields": [ + "allowed_outputs", + "assistant_memory", + "prompt_context", + "runtime_exclusivity", + "websocket_api_contracts" + ], + "replay": [ + "start_autonomous(lean_enabled=True, papers=True, proofs=True)", + "start_manual_compiler()" + ], + "result": "passed", + "scenario_id": "generated_runtime_start_exclusivity_seed_29", + "schema_version": 1, + "seed": 29, + "target_artifact": "targets/generated_runtime_start_exclusivity.json", + "target_id": "generated_runtime_start_exclusivity" +} diff --git a/tests/workflow_cross_field/results/generated_runtime_start_exclusivity_seed_7.json b/tests/workflow_cross_field/results/generated_runtime_start_exclusivity_seed_7.json new file mode 100644 index 0000000..1ed0325 --- /dev/null +++ b/tests/workflow_cross_field/results/generated_runtime_start_exclusivity_seed_7.json @@ -0,0 +1,40 @@ +{ + "action_ids": [ + "start_leanoj", + "attempt_conflicting_start" + ], + "adapter": "model", + "artifact_type": "result", + "diagnostics": { + "final_mode": "leanoj", + "final_phase": "leanoj_brainstorm" + }, + "evidence": [ + "start_blocked" + ], + "exercised_invariants": [ + "runtime.single_active_workflow" + ], + "fields": [ + "runtime_exclusivity", + "websocket_api_contracts" + ], + "invariants": [ + "runtime.single_active_workflow" + ], + "observed_fields": [ + "runtime_exclusivity", + "websocket_api_contracts", + "workflow_filesystem_state" + ], + "replay": [ + "start_leanoj()", + "start_manual_compiler()" + ], + "result": "passed", + "scenario_id": "generated_runtime_start_exclusivity_seed_7", + "schema_version": 1, + "seed": 7, + "target_artifact": "targets/generated_runtime_start_exclusivity.json", + "target_id": "generated_runtime_start_exclusivity" +} diff --git a/tests/workflow_cross_field/results/generated_scoped_registered_proof_events_seed_29.json b/tests/workflow_cross_field/results/generated_scoped_registered_proof_events_seed_29.json new file mode 100644 index 0000000..a6a4f3e --- /dev/null +++ b/tests/workflow_cross_field/results/generated_scoped_registered_proof_events_seed_29.json @@ -0,0 +1,47 @@ +{ + "action_ids": [ + "start_autonomous_papers_and_proofs", + "emit_registered_proof_verified" + ], + "adapter": "model", + "artifact_type": "result", + "diagnostics": { + "final_mode": "autonomous", + "final_phase": "topic_exploration" + }, + "evidence": [ + "registered_proof_event", + "scoped_event" + ], + "exercised_invariants": [ + "events.frontend_events_include_scope_phase", + "events.proof_verified_after_registration", + "proof_scope.autonomous_events_not_manual" + ], + "fields": [ + "websocket_api_contracts", + "proof_scope_isolation" + ], + "invariants": [ + "events.frontend_events_include_scope_phase", + "events.proof_verified_after_registration", + "proof_scope.autonomous_events_not_manual" + ], + "observed_fields": [ + "allowed_outputs", + "assistant_memory", + "prompt_context", + "proof_scope_isolation", + "websocket_api_contracts" + ], + "replay": [ + "start_autonomous(lean_enabled=True, papers=True, proofs=True)", + "emit_registered_proof_verified(proof_id='registered-proof-1')" + ], + "result": "passed", + "scenario_id": "generated_scoped_registered_proof_events_seed_29", + "schema_version": 1, + "seed": 29, + "target_artifact": "targets/generated_scoped_registered_proof_events.json", + "target_id": "generated_scoped_registered_proof_events" +} diff --git a/tests/workflow_cross_field/results/generated_scoped_registered_proof_events_seed_7.json b/tests/workflow_cross_field/results/generated_scoped_registered_proof_events_seed_7.json new file mode 100644 index 0000000..9a33c6c --- /dev/null +++ b/tests/workflow_cross_field/results/generated_scoped_registered_proof_events_seed_7.json @@ -0,0 +1,47 @@ +{ + "action_ids": [ + "start_autonomous_papers_and_proofs", + "emit_registered_proof_verified" + ], + "adapter": "model", + "artifact_type": "result", + "diagnostics": { + "final_mode": "autonomous", + "final_phase": "topic_exploration" + }, + "evidence": [ + "registered_proof_event", + "scoped_event" + ], + "exercised_invariants": [ + "events.frontend_events_include_scope_phase", + "events.proof_verified_after_registration", + "proof_scope.autonomous_events_not_manual" + ], + "fields": [ + "websocket_api_contracts", + "proof_scope_isolation" + ], + "invariants": [ + "events.frontend_events_include_scope_phase", + "events.proof_verified_after_registration", + "proof_scope.autonomous_events_not_manual" + ], + "observed_fields": [ + "allowed_outputs", + "assistant_memory", + "prompt_context", + "proof_scope_isolation", + "websocket_api_contracts" + ], + "replay": [ + "start_autonomous(lean_enabled=True, papers=True, proofs=True)", + "emit_registered_proof_verified(proof_id='registered-proof-1')" + ], + "result": "passed", + "scenario_id": "generated_scoped_registered_proof_events_seed_7", + "schema_version": 1, + "seed": 7, + "target_artifact": "targets/generated_scoped_registered_proof_events.json", + "target_id": "generated_scoped_registered_proof_events" +} diff --git a/tests/workflow_cross_field/results/model_allowed_outputs_provider_pause_stop_resume.json b/tests/workflow_cross_field/results/model_allowed_outputs_provider_pause_stop_resume.json new file mode 100644 index 0000000..a6035f1 --- /dev/null +++ b/tests/workflow_cross_field/results/model_allowed_outputs_provider_pause_stop_resume.json @@ -0,0 +1,36 @@ +{ + "adapter": "model", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "provider_pause", + "stop_resume", + "reset_credit" + ], + "fields": [ + "allowed_outputs", + "provider_pause_resume", + "workflow_filesystem_state" + ], + "invariants": [ + "outputs.proofs_only_no_paper_phase", + "provider.pause_preserves_checkpoint", + "provider.stop_resume_preserves_pause", + "provider.reset_wakes_without_corrupting_checkpoint" + ], + "replay": [ + "start_autonomous_proofs_only", + "complete_topic_exploration", + "simulate_credit_exhaustion", + "complete_brainstorm", + "stop", + "resume", + "reset_credit" + ], + "result": "passed", + "scenario_id": "model_allowed_outputs_provider_pause_stop_resume", + "schema_version": 1, + "seed": null, + "source": "curated", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" +} diff --git a/tests/workflow_cross_field/results/model_assistant_prompt_validator_exclusion.json b/tests/workflow_cross_field/results/model_assistant_prompt_validator_exclusion.json new file mode 100644 index 0000000..a687023 --- /dev/null +++ b/tests/workflow_cross_field/results/model_assistant_prompt_validator_exclusion.json @@ -0,0 +1,31 @@ +{ + "adapter": "model", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "assistant_pack", + "assistant_disable", + "prompt_context" + ], + "fields": [ + "assistant_memory", + "prompt_context" + ], + "invariants": [ + "assistant.no_validator_injection", + "assistant.disable_clears_live_pack", + "prompt.validator_excludes_assistant_memory" + ], + "replay": [ + "start_autonomous_papers_and_proofs", + "refresh_assistant_pack", + "prepare_prompt_context", + "disable_session_history" + ], + "result": "passed", + "scenario_id": "model_assistant_prompt_validator_exclusion", + "schema_version": 1, + "seed": null, + "source": "curated", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" +} diff --git a/tests/workflow_cross_field/results/model_manual_proof_clear_scope_assistant.json b/tests/workflow_cross_field/results/model_manual_proof_clear_scope_assistant.json new file mode 100644 index 0000000..76390b1 --- /dev/null +++ b/tests/workflow_cross_field/results/model_manual_proof_clear_scope_assistant.json @@ -0,0 +1,34 @@ +{ + "adapter": "model", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "manual_proof", + "manual_archive", + "assistant_pack" + ], + "fields": [ + "proof_scope_isolation", + "workflow_filesystem_state", + "assistant_memory" + ], + "invariants": [ + "proof_scope.manual_not_in_autonomous_current", + "proof_scope.manual_clear_archives_active", + "state.clear_removes_active_preserves_history", + "assistant.disable_clears_live_pack" + ], + "replay": [ + "start_manual_compiler", + "enable_manual_lean", + "run_manual_proof_check", + "refresh_assistant_pack", + "clear" + ], + "result": "passed", + "scenario_id": "model_manual_proof_clear_scope_assistant", + "schema_version": 1, + "seed": null, + "source": "curated", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" +} diff --git a/tests/workflow_cross_field/results/model_proof_runtime_gating_allowed_outputs.json b/tests/workflow_cross_field/results/model_proof_runtime_gating_allowed_outputs.json new file mode 100644 index 0000000..68ac1aa --- /dev/null +++ b/tests/workflow_cross_field/results/model_proof_runtime_gating_allowed_outputs.json @@ -0,0 +1,31 @@ +{ + "adapter": "model", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "no_outputs_rejected", + "papers_only" + ], + "fields": [ + "proof_runtime_gating", + "allowed_outputs" + ], + "invariants": [ + "outputs.at_least_one_output_enabled", + "outputs.papers_only_skips_proof_work", + "proof_runtime.no_lean_when_disabled", + "proof_runtime.no_smt_when_disabled" + ], + "replay": [ + "start_autonomous_no_outputs", + "start_autonomous_papers_only", + "complete_topic_exploration", + "complete_brainstorm" + ], + "result": "passed", + "scenario_id": "model_proof_runtime_gating_allowed_outputs", + "schema_version": 1, + "seed": null, + "source": "curated", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" +} diff --git a/tests/workflow_cross_field/results/model_websocket_proof_scope_events.json b/tests/workflow_cross_field/results/model_websocket_proof_scope_events.json new file mode 100644 index 0000000..bb86c0e --- /dev/null +++ b/tests/workflow_cross_field/results/model_websocket_proof_scope_events.json @@ -0,0 +1,29 @@ +{ + "adapter": "model", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "scoped_event", + "registered_proof_event" + ], + "fields": [ + "websocket_api_contracts", + "proof_scope_isolation" + ], + "invariants": [ + "events.frontend_events_include_scope_phase", + "events.proof_verified_after_registration", + "proof_scope.autonomous_events_not_manual" + ], + "replay": [ + "start_autonomous_papers_and_proofs", + "emit_frontend_scoped_event", + "emit_registered_proof_verified" + ], + "result": "passed", + "scenario_id": "model_websocket_proof_scope_events", + "schema_version": 1, + "seed": null, + "source": "curated", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" +} diff --git a/tests/workflow_cross_field/results/real_actual_route_start_conflict_matrix_no_side_effects.json b/tests/workflow_cross_field/results/real_actual_route_start_conflict_matrix_no_side_effects.json new file mode 100644 index 0000000..fde77c9 --- /dev/null +++ b/tests/workflow_cross_field/results/real_actual_route_start_conflict_matrix_no_side_effects.json @@ -0,0 +1,25 @@ +{ + "adapter": "real_route", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "sixteen_actual_route_calls", + "pending_owner", + "no_initialize_side_effects" + ], + "fields": [ + "runtime_exclusivity", + "workflow_filesystem_state" + ], + "invariants": [ + "runtime.single_active_workflow", + "runtime.child_tasks_count_as_active" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_actual_route_start_conflict_matrix_no_side_effects", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" +} diff --git a/tests/workflow_cross_field/results/real_autonomous_overflow_terminal_stop_once.json b/tests/workflow_cross_field/results/real_autonomous_overflow_terminal_stop_once.json new file mode 100644 index 0000000..b8809f8 --- /dev/null +++ b/tests/workflow_cross_field/results/real_autonomous_overflow_terminal_stop_once.json @@ -0,0 +1,25 @@ +{ + "adapter": "real_coordinator", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "configured_effective_route", + "single_terminal_stop" + ], + "fields": [ + "websocket_api_contracts", + "workflow_filesystem_state", + "runtime_exclusivity" + ], + "invariants": [ + "events.context_overflow_route_identity", + "events.context_overflow_terminal_stop_once" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_autonomous_overflow_terminal_stop_once", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_build_d_overflow_lifecycles.py" +} diff --git a/tests/workflow_cross_field/results/real_autonomous_proofs_only_handoff_returns_to_topic_exploration.json b/tests/workflow_cross_field/results/real_autonomous_proofs_only_handoff_returns_to_topic_exploration.json new file mode 100644 index 0000000..ace6405 --- /dev/null +++ b/tests/workflow_cross_field/results/real_autonomous_proofs_only_handoff_returns_to_topic_exploration.json @@ -0,0 +1,24 @@ +{ + "adapter": "real_coordinator", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "proofs_only_handoff", + "topic_exploration" + ], + "fields": [ + "allowed_outputs", + "proof_runtime_gating", + "workflow_filesystem_state" + ], + "invariants": [ + "outputs.proofs_only_no_paper_phase" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_autonomous_proofs_only_handoff_returns_to_topic_exploration", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_proof_routes.py" +} diff --git a/tests/workflow_cross_field/results/real_autonomous_provider_credit_pause_preserves_checkpoint_and_resumes.json b/tests/workflow_cross_field/results/real_autonomous_provider_credit_pause_preserves_checkpoint_and_resumes.json new file mode 100644 index 0000000..af69b02 --- /dev/null +++ b/tests/workflow_cross_field/results/real_autonomous_provider_credit_pause_preserves_checkpoint_and_resumes.json @@ -0,0 +1,24 @@ +{ + "adapter": "real_coordinator", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "provider_pause", + "checkpoint_resume" + ], + "fields": [ + "provider_pause_resume", + "workflow_filesystem_state", + "proof_runtime_gating" + ], + "invariants": [ + "provider.pause_preserves_checkpoint" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_autonomous_provider_credit_pause_preserves_checkpoint_and_resumes", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_proof_routes.py" +} diff --git a/tests/workflow_cross_field/results/real_direct_source_rag_exclusion_matrix.json b/tests/workflow_cross_field/results/real_direct_source_rag_exclusion_matrix.json new file mode 100644 index 0000000..49db2cd --- /dev/null +++ b/tests/workflow_cross_field/results/real_direct_source_rag_exclusion_matrix.json @@ -0,0 +1,25 @@ +{ + "adapter": "real_coordinator", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "aggregator_direct_and_offloaded_sources", + "compiler_construction_exclusions", + "compiler_rigor_exclusions", + "offloaded_source_retrievable" + ], + "fields": [ + "prompt_context", + "workflow_filesystem_state" + ], + "invariants": [ + "prompt.direct_sources_excluded_from_rag" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_direct_source_rag_exclusion_matrix", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_build_e_rag_prompt_context.py" +} diff --git a/tests/workflow_cross_field/results/real_disabled_proof_runtime_skips_autonomous_checkpoint.json b/tests/workflow_cross_field/results/real_disabled_proof_runtime_skips_autonomous_checkpoint.json new file mode 100644 index 0000000..152140e --- /dev/null +++ b/tests/workflow_cross_field/results/real_disabled_proof_runtime_skips_autonomous_checkpoint.json @@ -0,0 +1,24 @@ +{ + "adapter": "real_coordinator", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "disabled_runtime", + "zero_proof_stage_calls" + ], + "fields": [ + "proof_runtime_gating", + "allowed_outputs", + "workflow_filesystem_state" + ], + "invariants": [ + "proof_runtime.no_lean_when_disabled" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_disabled_proof_runtime_skips_autonomous_checkpoint", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_high_value_scenarios.py" +} diff --git a/tests/workflow_cross_field/results/real_generated_proof_appendix_stripping.json b/tests/workflow_cross_field/results/real_generated_proof_appendix_stripping.json new file mode 100644 index 0000000..2da89b8 --- /dev/null +++ b/tests/workflow_cross_field/results/real_generated_proof_appendix_stripping.json @@ -0,0 +1,23 @@ +{ + "adapter": "real_coordinator", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "brainstorm_appendix_stripped", + "paper_appendix_stripped" + ], + "fields": [ + "prompt_context", + "proof_scope_isolation" + ], + "invariants": [ + "proof_scope.generated_appendices_stripped_from_prompts" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_generated_proof_appendix_stripping", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" +} diff --git a/tests/workflow_cross_field/results/real_hosted_proof_settings_returns_desktop_unavailable.json b/tests/workflow_cross_field/results/real_hosted_proof_settings_returns_desktop_unavailable.json new file mode 100644 index 0000000..f7e8587 --- /dev/null +++ b/tests/workflow_cross_field/results/real_hosted_proof_settings_returns_desktop_unavailable.json @@ -0,0 +1,24 @@ +{ + "adapter": "real_route", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "hosted_mode", + "http_501" + ], + "fields": [ + "proof_runtime_gating", + "websocket_api_contracts", + "allowed_outputs" + ], + "invariants": [ + "proof_runtime.hosted_proof_settings_unavailable" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_hosted_proof_settings_returns_desktop_unavailable", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_high_value_scenarios.py" +} diff --git a/tests/workflow_cross_field/results/real_leanoj_coordinator_skip_force_clear_and_intermediate_edit.json b/tests/workflow_cross_field/results/real_leanoj_coordinator_skip_force_clear_and_intermediate_edit.json new file mode 100644 index 0000000..6d7cd44 --- /dev/null +++ b/tests/workflow_cross_field/results/real_leanoj_coordinator_skip_force_clear_and_intermediate_edit.json @@ -0,0 +1,26 @@ +{ + "adapter": "real_coordinator", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "real_skip_consume", + "real_force_consume", + "state_preserved_then_cleared", + "master_proof_temp_root" + ], + "fields": [ + "runtime_exclusivity", + "workflow_filesystem_state", + "proof_scope_isolation" + ], + "invariants": [ + "state.runtime_roots_are_active_roots" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_leanoj_coordinator_skip_force_clear_and_intermediate_edit", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" +} diff --git a/tests/workflow_cross_field/results/real_leanoj_full_final_loop_not_safely_bounded.json b/tests/workflow_cross_field/results/real_leanoj_full_final_loop_not_safely_bounded.json new file mode 100644 index 0000000..bd57bfa --- /dev/null +++ b/tests/workflow_cross_field/results/real_leanoj_full_final_loop_not_safely_bounded.json @@ -0,0 +1,23 @@ +{ + "adapter": "real_coordinator", + "artifact_type": "result", + "diagnostics": { + "reason": "The production final loop combines unbounded model iteration, Lean execution, semantic review, persistence, and registration without a bounded route-level seam. Direct registration ordering is covered elsewhere; publishing a full lifecycle pass would require faking the transition owner rather than observing it." + }, + "evidence": [], + "fields": [ + "proof_runtime_gating", + "workflow_filesystem_state", + "websocket_api_contracts" + ], + "invariants": [ + "events.proof_verified_after_registration" + ], + "replay": [], + "result": "blocked", + "scenario_id": "real_leanoj_full_final_loop_not_safely_bounded", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" +} diff --git a/tests/workflow_cross_field/results/real_leanoj_master_proof_stop_resume_preserves_isolated_state.json b/tests/workflow_cross_field/results/real_leanoj_master_proof_stop_resume_preserves_isolated_state.json new file mode 100644 index 0000000..5610a33 --- /dev/null +++ b/tests/workflow_cross_field/results/real_leanoj_master_proof_stop_resume_preserves_isolated_state.json @@ -0,0 +1,24 @@ +{ + "adapter": "real_coordinator", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "master_proof_persisted", + "session_isolation" + ], + "fields": [ + "workflow_filesystem_state", + "proof_scope_isolation", + "runtime_exclusivity" + ], + "invariants": [ + "proof_scope.autonomous_events_not_manual" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_leanoj_master_proof_stop_resume_preserves_isolated_state", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_proof_routes.py" +} diff --git a/tests/workflow_cross_field/results/real_manual_aggregator_clear_archives_before_clear.json b/tests/workflow_cross_field/results/real_manual_aggregator_clear_archives_before_clear.json new file mode 100644 index 0000000..ae1e154 --- /dev/null +++ b/tests/workflow_cross_field/results/real_manual_aggregator_clear_archives_before_clear.json @@ -0,0 +1,23 @@ +{ + "adapter": "real_route", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "archive_before_clear", + "upload_cleanup" + ], + "fields": [ + "proof_scope_isolation", + "workflow_filesystem_state" + ], + "invariants": [ + "proof_scope.manual_clear_archives_active" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_manual_aggregator_clear_archives_before_clear", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" +} diff --git a/tests/workflow_cross_field/results/real_manual_aggregator_durable_hydration_and_clear_blocker.json b/tests/workflow_cross_field/results/real_manual_aggregator_durable_hydration_and_clear_blocker.json new file mode 100644 index 0000000..8eb0046 --- /dev/null +++ b/tests/workflow_cross_field/results/real_manual_aggregator_durable_hydration_and_clear_blocker.json @@ -0,0 +1,25 @@ +{ + "adapter": "real_coordinator", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "persisted_result_hydration", + "temp_root", + "blocked_clear_no_archive" + ], + "fields": [ + "workflow_filesystem_state", + "proof_scope_isolation", + "prompt_context" + ], + "invariants": [ + "state.runtime_roots_are_active_roots" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_manual_aggregator_durable_hydration_and_clear_blocker", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" +} diff --git a/tests/workflow_cross_field/results/real_manual_aggregator_overflow_identity_persists_across_reload.json b/tests/workflow_cross_field/results/real_manual_aggregator_overflow_identity_persists_across_reload.json new file mode 100644 index 0000000..8ce1884 --- /dev/null +++ b/tests/workflow_cross_field/results/real_manual_aggregator_overflow_identity_persists_across_reload.json @@ -0,0 +1,26 @@ +{ + "adapter": "real_coordinator", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "configured_effective_route", + "event_log_reload", + "temp_root" + ], + "fields": [ + "websocket_api_contracts", + "workflow_filesystem_state", + "provider_pause_resume" + ], + "invariants": [ + "events.context_overflow_route_identity", + "events.context_overflow_persists_across_reload" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_manual_aggregator_overflow_identity_persists_across_reload", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_build_d_overflow_lifecycles.py" +} diff --git a/tests/workflow_cross_field/results/real_manual_aggregator_route_lifecycle_output_and_temp_root.json b/tests/workflow_cross_field/results/real_manual_aggregator_route_lifecycle_output_and_temp_root.json new file mode 100644 index 0000000..3923d76 --- /dev/null +++ b/tests/workflow_cross_field/results/real_manual_aggregator_route_lifecycle_output_and_temp_root.json @@ -0,0 +1,25 @@ +{ + "adapter": "real_route", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "route_start_stop", + "saved_output", + "temp_root_containment" + ], + "fields": [ + "workflow_filesystem_state", + "runtime_exclusivity", + "prompt_context" + ], + "invariants": [ + "runtime.single_active_workflow" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_manual_aggregator_route_lifecycle_output_and_temp_root", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" +} diff --git a/tests/workflow_cross_field/results/real_manual_archive_and_leanoj_clear_preserve_scope_contract.json b/tests/workflow_cross_field/results/real_manual_archive_and_leanoj_clear_preserve_scope_contract.json new file mode 100644 index 0000000..21ef104 --- /dev/null +++ b/tests/workflow_cross_field/results/real_manual_archive_and_leanoj_clear_preserve_scope_contract.json @@ -0,0 +1,26 @@ +{ + "adapter": "real_coordinator", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "physical_manual_archive", + "active_store_empty", + "leanoj_root_removed" + ], + "fields": [ + "proof_scope_isolation", + "workflow_filesystem_state", + "assistant_memory" + ], + "invariants": [ + "proof_scope.manual_clear_archives_active", + "state.clear_removes_active_preserves_history" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_manual_archive_and_leanoj_clear_preserve_scope_contract", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_build_d_proof_scope_filesystem.py" +} diff --git a/tests/workflow_cross_field/results/real_manual_compiler_papers_lifecycle_and_clear_archive.json b/tests/workflow_cross_field/results/real_manual_compiler_papers_lifecycle_and_clear_archive.json new file mode 100644 index 0000000..98bebc8 --- /dev/null +++ b/tests/workflow_cross_field/results/real_manual_compiler_papers_lifecycle_and_clear_archive.json @@ -0,0 +1,26 @@ +{ + "adapter": "real_route", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "papers_only_start_stop", + "archive_before_appendix_and_paper_clear", + "temp_archive_root" + ], + "fields": [ + "allowed_outputs", + "workflow_filesystem_state", + "proof_scope_isolation" + ], + "invariants": [ + "proof_scope.manual_clear_archives_active", + "state.clear_removes_active_preserves_history" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_manual_compiler_papers_lifecycle_and_clear_archive", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" +} diff --git a/tests/workflow_cross_field/results/real_manual_compiler_proof_only_background_ownership.json b/tests/workflow_cross_field/results/real_manual_compiler_proof_only_background_ownership.json new file mode 100644 index 0000000..80bfcf4 --- /dev/null +++ b/tests/workflow_cross_field/results/real_manual_compiler_proof_only_background_ownership.json @@ -0,0 +1,25 @@ +{ + "adapter": "real_route", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "actual_proof_only_start_route", + "live_background_task", + "second_start_conflict" + ], + "fields": [ + "allowed_outputs", + "runtime_exclusivity", + "proof_runtime_gating" + ], + "invariants": [ + "runtime.child_tasks_count_as_active" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_manual_compiler_proof_only_background_ownership", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" +} diff --git a/tests/workflow_cross_field/results/real_manual_compiler_proof_only_uses_rigor_settings_and_manual_scope.json b/tests/workflow_cross_field/results/real_manual_compiler_proof_only_uses_rigor_settings_and_manual_scope.json new file mode 100644 index 0000000..348d5fc --- /dev/null +++ b/tests/workflow_cross_field/results/real_manual_compiler_proof_only_uses_rigor_settings_and_manual_scope.json @@ -0,0 +1,24 @@ +{ + "adapter": "real_coordinator", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "manual_scope", + "rigor_settings" + ], + "fields": [ + "allowed_outputs", + "proof_scope_isolation", + "prompt_context" + ], + "invariants": [ + "proof_scope.manual_not_in_autonomous_current" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_manual_compiler_proof_only_uses_rigor_settings_and_manual_scope", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_proof_routes.py" +} diff --git a/tests/workflow_cross_field/results/real_manual_compiler_rejects_no_allowed_outputs.json b/tests/workflow_cross_field/results/real_manual_compiler_rejects_no_allowed_outputs.json new file mode 100644 index 0000000..007e9d7 --- /dev/null +++ b/tests/workflow_cross_field/results/real_manual_compiler_rejects_no_allowed_outputs.json @@ -0,0 +1,23 @@ +{ + "adapter": "real_route", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "http_400", + "coordinator_not_started" + ], + "fields": [ + "allowed_outputs", + "runtime_exclusivity" + ], + "invariants": [ + "outputs.at_least_one_output_enabled" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_manual_compiler_rejects_no_allowed_outputs", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" +} diff --git a/tests/workflow_cross_field/results/real_model_visible_prompts_strip_generated_proof_appendices.json b/tests/workflow_cross_field/results/real_model_visible_prompts_strip_generated_proof_appendices.json new file mode 100644 index 0000000..92ca6b5 --- /dev/null +++ b/tests/workflow_cross_field/results/real_model_visible_prompts_strip_generated_proof_appendices.json @@ -0,0 +1,27 @@ +{ + "adapter": "real_coordinator", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "paper_head_and_tail_preserved", + "generated_appendix_absent", + "verified_library_context_present", + "rigor_message_captured" + ], + "fields": [ + "prompt_context", + "proof_scope_isolation", + "proof_runtime_gating" + ], + "invariants": [ + "prompt.generated_appendices_stripped", + "proof_scope.generated_appendices_stripped_from_prompts" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_model_visible_prompts_strip_generated_proof_appendices", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_build_e_prompt_boundaries.py" +} diff --git a/tests/workflow_cross_field/results/real_parent_action_fencing_unavailable_without_production_seam.json b/tests/workflow_cross_field/results/real_parent_action_fencing_unavailable_without_production_seam.json new file mode 100644 index 0000000..a16db7e --- /dev/null +++ b/tests/workflow_cross_field/results/real_parent_action_fencing_unavailable_without_production_seam.json @@ -0,0 +1,23 @@ +{ + "adapter": "real_coordinator", + "artifact_type": "result", + "diagnostics": { + "reason": "The invariant catalog declares model-only support and no existing isolated production seam exposes stale child output acceptance after parent takeover." + }, + "evidence": [], + "fields": [ + "runtime_exclusivity", + "workflow_filesystem_state", + "websocket_api_contracts" + ], + "invariants": [ + "runtime.parent_action_fences_child_outputs" + ], + "replay": [], + "result": "blocked", + "scenario_id": "real_parent_action_fencing_unavailable_without_production_seam", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_high_value_scenarios.py" +} diff --git a/tests/workflow_cross_field/results/real_proof_context_overflow_is_scoped_nonfatal.json b/tests/workflow_cross_field/results/real_proof_context_overflow_is_scoped_nonfatal.json new file mode 100644 index 0000000..6a1d44a --- /dev/null +++ b/tests/workflow_cross_field/results/real_proof_context_overflow_is_scoped_nonfatal.json @@ -0,0 +1,26 @@ +{ + "adapter": "real_coordinator", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "configured_effective_route", + "fatal_false", + "no_parent_stop" + ], + "fields": [ + "websocket_api_contracts", + "proof_runtime_gating", + "provider_pause_resume" + ], + "invariants": [ + "events.context_overflow_route_identity", + "events.proof_context_overflow_nonfatal" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_proof_context_overflow_is_scoped_nonfatal", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_build_d_overflow_lifecycles.py" +} diff --git a/tests/workflow_cross_field/results/real_proof_scope_matrix_isolated_under_temp_root.json b/tests/workflow_cross_field/results/real_proof_scope_matrix_isolated_under_temp_root.json new file mode 100644 index 0000000..279f61f --- /dev/null +++ b/tests/workflow_cross_field/results/real_proof_scope_matrix_isolated_under_temp_root.json @@ -0,0 +1,26 @@ +{ + "adapter": "real_route", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "autonomous_current_and_history", + "manual_active_and_history", + "leanoj_current_and_history", + "pairwise_sentinels" + ], + "fields": [ + "proof_scope_isolation", + "workflow_filesystem_state", + "websocket_api_contracts" + ], + "invariants": [ + "proof_scope.manual_not_in_autonomous_current" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_proof_scope_matrix_isolated_under_temp_root", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_build_d_proof_scope_filesystem.py" +} diff --git a/tests/workflow_cross_field/results/real_provider_stop_reset_checkpoint_unavailable_without_wait_seam.json b/tests/workflow_cross_field/results/real_provider_stop_reset_checkpoint_unavailable_without_wait_seam.json new file mode 100644 index 0000000..118cb25 --- /dev/null +++ b/tests/workflow_cross_field/results/real_provider_stop_reset_checkpoint_unavailable_without_wait_seam.json @@ -0,0 +1,23 @@ +{ + "adapter": "real_coordinator", + "artifact_type": "result", + "diagnostics": { + "reason": "Existing coordinator coverage observes pause/reset resume, but no bounded test seam independently interleaves user Stop while the provider wait is active." + }, + "evidence": [], + "fields": [ + "provider_pause_resume", + "workflow_filesystem_state", + "runtime_exclusivity" + ], + "invariants": [ + "provider.stop_resume_preserves_pause" + ], + "replay": [], + "result": "blocked", + "scenario_id": "real_provider_stop_reset_checkpoint_unavailable_without_wait_seam", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_high_value_scenarios.py" +} diff --git a/tests/workflow_cross_field/results/real_pruned_paper_preserved_but_removed_from_active_context.json b/tests/workflow_cross_field/results/real_pruned_paper_preserved_but_removed_from_active_context.json new file mode 100644 index 0000000..7c81b82 --- /dev/null +++ b/tests/workflow_cross_field/results/real_pruned_paper_preserved_but_removed_from_active_context.json @@ -0,0 +1,26 @@ +{ + "adapter": "real_route", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "pruned_history_downloadable", + "active_enumeration_empty", + "brainstorm_reference_removed", + "rag_source_removed", + "temp_root" + ], + "fields": [ + "workflow_filesystem_state", + "prompt_context" + ], + "invariants": [ + "state.pruned_papers_excluded_from_context" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_pruned_paper_preserved_but_removed_from_active_context", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_build_d_proof_scope_filesystem.py" +} diff --git a/tests/workflow_cross_field/results/real_rigor_mandatory_source_overflow_fails_before_model_or_rag.json b/tests/workflow_cross_field/results/real_rigor_mandatory_source_overflow_fails_before_model_or_rag.json new file mode 100644 index 0000000..6f3baf9 --- /dev/null +++ b/tests/workflow_cross_field/results/real_rigor_mandatory_source_overflow_fails_before_model_or_rag.json @@ -0,0 +1,26 @@ +{ + "adapter": "real_coordinator", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "visible_value_error", + "zero_model_calls", + "zero_rag_calls", + "source_unchanged" + ], + "fields": [ + "prompt_context", + "proof_runtime_gating", + "websocket_api_contracts" + ], + "invariants": [ + "prompt.mandatory_source_overflow_fails_visible" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_rigor_mandatory_source_overflow_fails_before_model_or_rag", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_build_e_prompt_boundaries.py" +} diff --git a/tests/workflow_cross_field/results/real_route_pending_child_activity_counts_as_active.json b/tests/workflow_cross_field/results/real_route_pending_child_activity_counts_as_active.json new file mode 100644 index 0000000..74b1443 --- /dev/null +++ b/tests/workflow_cross_field/results/real_route_pending_child_activity_counts_as_active.json @@ -0,0 +1,23 @@ +{ + "adapter": "real_route", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "pending_child_activity", + "route_conflict_response" + ], + "fields": [ + "runtime_exclusivity", + "workflow_filesystem_state" + ], + "invariants": [ + "runtime.single_active_workflow" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_route_pending_child_activity_counts_as_active", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_route_conflicts.py" +} diff --git a/tests/workflow_cross_field/results/real_route_start_conflicts_preserve_single_workflow.json b/tests/workflow_cross_field/results/real_route_start_conflicts_preserve_single_workflow.json new file mode 100644 index 0000000..6646e1c --- /dev/null +++ b/tests/workflow_cross_field/results/real_route_start_conflicts_preserve_single_workflow.json @@ -0,0 +1,23 @@ +{ + "adapter": "real_route", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "route_conflict_response", + "single_active_workflow" + ], + "fields": [ + "runtime_exclusivity", + "websocket_api_contracts" + ], + "invariants": [ + "runtime.single_active_workflow" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_route_start_conflicts_preserve_single_workflow", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_route_conflicts.py" +} diff --git a/tests/workflow_cross_field/results/real_shared_start_guard_representative_race.json b/tests/workflow_cross_field/results/real_shared_start_guard_representative_race.json new file mode 100644 index 0000000..0859469 --- /dev/null +++ b/tests/workflow_cross_field/results/real_shared_start_guard_representative_race.json @@ -0,0 +1,23 @@ +{ + "adapter": "real_route", + "artifact_type": "result", + "diagnostics": {}, + "evidence": [ + "concurrent_start_race", + "one_winner" + ], + "fields": [ + "runtime_exclusivity", + "websocket_api_contracts" + ], + "invariants": [ + "runtime.single_active_workflow" + ], + "replay": [], + "result": "passed", + "scenario_id": "real_shared_start_guard_representative_race", + "schema_version": 1, + "seed": null, + "source": "real_adapter", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" +} diff --git a/tests/workflow_cross_field/scenarios/model_allowed_outputs_provider_pause_stop_resume.json b/tests/workflow_cross_field/scenarios/model_allowed_outputs_provider_pause_stop_resume.json new file mode 100644 index 0000000..43ebf1f --- /dev/null +++ b/tests/workflow_cross_field/scenarios/model_allowed_outputs_provider_pause_stop_resume.json @@ -0,0 +1,32 @@ +{ + "actions": [ + "start_autonomous_proofs_only", + "complete_topic_exploration", + "simulate_credit_exhaustion", + "complete_brainstorm", + "stop", + "resume", + "reset_credit" + ], + "adapter": "model", + "artifact_type": "scenario", + "expected_result": "passed", + "fields": [ + "allowed_outputs", + "provider_pause_resume", + "workflow_filesystem_state" + ], + "invariants": [ + "outputs.proofs_only_no_paper_phase", + "provider.pause_preserves_checkpoint", + "provider.stop_resume_preserves_pause", + "provider.reset_wakes_without_corrupting_checkpoint" + ], + "must_exercise": [ + "provider_pause", + "stop_resume", + "reset_credit" + ], + "scenario_id": "model_allowed_outputs_provider_pause_stop_resume", + "schema_version": 1 +} diff --git a/tests/workflow_cross_field/scenarios/model_assistant_prompt_validator_exclusion.json b/tests/workflow_cross_field/scenarios/model_assistant_prompt_validator_exclusion.json new file mode 100644 index 0000000..d77a113 --- /dev/null +++ b/tests/workflow_cross_field/scenarios/model_assistant_prompt_validator_exclusion.json @@ -0,0 +1,27 @@ +{ + "actions": [ + "start_autonomous_papers_and_proofs", + "refresh_assistant_pack", + "prepare_prompt_context", + "disable_session_history" + ], + "adapter": "model", + "artifact_type": "scenario", + "expected_result": "passed", + "fields": [ + "assistant_memory", + "prompt_context" + ], + "invariants": [ + "assistant.no_validator_injection", + "assistant.disable_clears_live_pack", + "prompt.validator_excludes_assistant_memory" + ], + "must_exercise": [ + "assistant_pack", + "assistant_disable", + "prompt_context" + ], + "scenario_id": "model_assistant_prompt_validator_exclusion", + "schema_version": 1 +} diff --git a/tests/workflow_cross_field/scenarios/model_manual_proof_clear_scope_assistant.json b/tests/workflow_cross_field/scenarios/model_manual_proof_clear_scope_assistant.json new file mode 100644 index 0000000..3479e56 --- /dev/null +++ b/tests/workflow_cross_field/scenarios/model_manual_proof_clear_scope_assistant.json @@ -0,0 +1,30 @@ +{ + "actions": [ + "start_manual_compiler", + "enable_manual_lean", + "run_manual_proof_check", + "refresh_assistant_pack", + "clear" + ], + "adapter": "model", + "artifact_type": "scenario", + "expected_result": "passed", + "fields": [ + "proof_scope_isolation", + "workflow_filesystem_state", + "assistant_memory" + ], + "invariants": [ + "proof_scope.manual_not_in_autonomous_current", + "proof_scope.manual_clear_archives_active", + "state.clear_removes_active_preserves_history", + "assistant.disable_clears_live_pack" + ], + "must_exercise": [ + "manual_proof", + "manual_archive", + "assistant_pack" + ], + "scenario_id": "model_manual_proof_clear_scope_assistant", + "schema_version": 1 +} diff --git a/tests/workflow_cross_field/scenarios/model_proof_runtime_gating_allowed_outputs.json b/tests/workflow_cross_field/scenarios/model_proof_runtime_gating_allowed_outputs.json new file mode 100644 index 0000000..60127aa --- /dev/null +++ b/tests/workflow_cross_field/scenarios/model_proof_runtime_gating_allowed_outputs.json @@ -0,0 +1,27 @@ +{ + "actions": [ + "start_autonomous_no_outputs", + "start_autonomous_papers_only", + "complete_topic_exploration", + "complete_brainstorm" + ], + "adapter": "model", + "artifact_type": "scenario", + "expected_result": "passed", + "fields": [ + "proof_runtime_gating", + "allowed_outputs" + ], + "invariants": [ + "outputs.at_least_one_output_enabled", + "outputs.papers_only_skips_proof_work", + "proof_runtime.no_lean_when_disabled", + "proof_runtime.no_smt_when_disabled" + ], + "must_exercise": [ + "no_outputs_rejected", + "papers_only" + ], + "scenario_id": "model_proof_runtime_gating_allowed_outputs", + "schema_version": 1 +} diff --git a/tests/workflow_cross_field/scenarios/model_websocket_proof_scope_events.json b/tests/workflow_cross_field/scenarios/model_websocket_proof_scope_events.json new file mode 100644 index 0000000..6836b64 --- /dev/null +++ b/tests/workflow_cross_field/scenarios/model_websocket_proof_scope_events.json @@ -0,0 +1,25 @@ +{ + "actions": [ + "start_autonomous_papers_and_proofs", + "emit_frontend_scoped_event", + "emit_registered_proof_verified" + ], + "adapter": "model", + "artifact_type": "scenario", + "expected_result": "passed", + "fields": [ + "websocket_api_contracts", + "proof_scope_isolation" + ], + "invariants": [ + "events.frontend_events_include_scope_phase", + "events.proof_verified_after_registration", + "proof_scope.autonomous_events_not_manual" + ], + "must_exercise": [ + "scoped_event", + "registered_proof_event" + ], + "scenario_id": "model_websocket_proof_scope_events", + "schema_version": 1 +} diff --git a/tests/workflow_cross_field/support_graph.json b/tests/workflow_cross_field/support_graph.json new file mode 100644 index 0000000..1a49bca --- /dev/null +++ b/tests/workflow_cross_field/support_graph.json @@ -0,0 +1,2629 @@ +{ + "artifact_type": "support_graph", + "basis": { + "declared": "Invariant catalog adapter declarations and target-risk requirements.", + "observed": "InteractionCoverage records produced by executable tests or truthful gap records." + }, + "edges": [ + { + "adapter": "browser_smoke", + "declared": false, + "field": "assistant_memory", + "invariant_id": "assistant.disable_clears_live_pack", + "observed": false, + "risk_id": "risk.assistant_memory_crosses_boundary", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "model", + "declared": true, + "field": "assistant_memory", + "invariant_id": "assistant.disable_clears_live_pack", + "observed": true, + "risk_id": "risk.assistant_memory_crosses_boundary", + "scenario_id": "generated_assistant_prompt_context_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "assistant_memory", + "invariant_id": "assistant.disable_clears_live_pack", + "observed": true, + "risk_id": "risk.assistant_memory_crosses_boundary", + "scenario_id": "generated_assistant_prompt_context_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "assistant_memory", + "invariant_id": "assistant.disable_clears_live_pack", + "observed": true, + "risk_id": "risk.assistant_memory_crosses_boundary", + "scenario_id": "model_assistant_prompt_validator_exclusion", + "status": "passed", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" + }, + { + "adapter": "model", + "declared": true, + "field": "assistant_memory", + "invariant_id": "assistant.disable_clears_live_pack", + "observed": true, + "risk_id": "risk.assistant_memory_crosses_boundary", + "scenario_id": "model_manual_proof_clear_scope_assistant", + "status": "passed", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" + }, + { + "adapter": "real_coordinator", + "declared": false, + "field": "assistant_memory", + "invariant_id": "assistant.disable_clears_live_pack", + "observed": false, + "risk_id": "risk.assistant_memory_crosses_boundary", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "real_route", + "declared": false, + "field": "assistant_memory", + "invariant_id": "assistant.disable_clears_live_pack", + "observed": false, + "risk_id": "risk.assistant_memory_crosses_boundary", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "browser_smoke", + "declared": false, + "field": "assistant_memory", + "invariant_id": "assistant.no_validator_injection", + "observed": false, + "risk_id": "risk.assistant_memory_crosses_boundary", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "model", + "declared": true, + "field": "assistant_memory", + "invariant_id": "assistant.no_validator_injection", + "observed": true, + "risk_id": "risk.assistant_memory_crosses_boundary", + "scenario_id": "generated_assistant_prompt_context_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "assistant_memory", + "invariant_id": "assistant.no_validator_injection", + "observed": true, + "risk_id": "risk.assistant_memory_crosses_boundary", + "scenario_id": "generated_assistant_prompt_context_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "assistant_memory", + "invariant_id": "assistant.no_validator_injection", + "observed": true, + "risk_id": "risk.assistant_memory_crosses_boundary", + "scenario_id": "model_assistant_prompt_validator_exclusion", + "status": "passed", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" + }, + { + "adapter": "real_coordinator", + "declared": false, + "field": "assistant_memory", + "invariant_id": "assistant.no_validator_injection", + "observed": false, + "risk_id": "risk.assistant_memory_crosses_boundary", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "real_route", + "declared": false, + "field": "assistant_memory", + "invariant_id": "assistant.no_validator_injection", + "observed": false, + "risk_id": "risk.assistant_memory_crosses_boundary", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "browser_smoke", + "declared": false, + "field": "prompt_context", + "invariant_id": "prompt.validator_excludes_assistant_memory", + "observed": false, + "risk_id": "risk.assistant_memory_crosses_boundary", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "model", + "declared": true, + "field": "prompt_context", + "invariant_id": "prompt.validator_excludes_assistant_memory", + "observed": true, + "risk_id": "risk.assistant_memory_crosses_boundary", + "scenario_id": "generated_assistant_prompt_context_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "prompt_context", + "invariant_id": "prompt.validator_excludes_assistant_memory", + "observed": true, + "risk_id": "risk.assistant_memory_crosses_boundary", + "scenario_id": "generated_assistant_prompt_context_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "prompt_context", + "invariant_id": "prompt.validator_excludes_assistant_memory", + "observed": true, + "risk_id": "risk.assistant_memory_crosses_boundary", + "scenario_id": "model_assistant_prompt_validator_exclusion", + "status": "passed", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" + }, + { + "adapter": "real_coordinator", + "declared": false, + "field": "prompt_context", + "invariant_id": "prompt.validator_excludes_assistant_memory", + "observed": false, + "risk_id": "risk.assistant_memory_crosses_boundary", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "real_route", + "declared": false, + "field": "prompt_context", + "invariant_id": "prompt.validator_excludes_assistant_memory", + "observed": false, + "risk_id": "risk.assistant_memory_crosses_boundary", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "browser_smoke", + "declared": false, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.manual_clear_archives_active", + "observed": false, + "risk_id": "risk.clear_destroys_history", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "model", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.manual_clear_archives_active", + "observed": true, + "risk_id": "risk.clear_destroys_history", + "scenario_id": "generated_manual_scope_clear_archive_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.manual_clear_archives_active", + "observed": true, + "risk_id": "risk.clear_destroys_history", + "scenario_id": "generated_manual_scope_clear_archive_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.manual_clear_archives_active", + "observed": true, + "risk_id": "risk.clear_destroys_history", + "scenario_id": "model_manual_proof_clear_scope_assistant", + "status": "passed", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" + }, + { + "adapter": "real_coordinator", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.manual_clear_archives_active", + "observed": true, + "risk_id": "risk.clear_destroys_history", + "scenario_id": "real_manual_archive_and_leanoj_clear_preserve_scope_contract", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_build_d_proof_scope_filesystem.py" + }, + { + "adapter": "real_route", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.manual_clear_archives_active", + "observed": true, + "risk_id": "risk.clear_destroys_history", + "scenario_id": "real_manual_aggregator_clear_archives_before_clear", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" + }, + { + "adapter": "real_route", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.manual_clear_archives_active", + "observed": true, + "risk_id": "risk.clear_destroys_history", + "scenario_id": "real_manual_compiler_papers_lifecycle_and_clear_archive", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" + }, + { + "adapter": "browser_smoke", + "declared": false, + "field": "workflow_filesystem_state", + "invariant_id": "state.clear_removes_active_preserves_history", + "observed": false, + "risk_id": "risk.clear_destroys_history", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "model", + "declared": true, + "field": "workflow_filesystem_state", + "invariant_id": "state.clear_removes_active_preserves_history", + "observed": true, + "risk_id": "risk.clear_destroys_history", + "scenario_id": "generated_manual_scope_clear_archive_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "workflow_filesystem_state", + "invariant_id": "state.clear_removes_active_preserves_history", + "observed": true, + "risk_id": "risk.clear_destroys_history", + "scenario_id": "generated_manual_scope_clear_archive_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "workflow_filesystem_state", + "invariant_id": "state.clear_removes_active_preserves_history", + "observed": true, + "risk_id": "risk.clear_destroys_history", + "scenario_id": "model_manual_proof_clear_scope_assistant", + "status": "passed", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" + }, + { + "adapter": "real_coordinator", + "declared": true, + "field": "workflow_filesystem_state", + "invariant_id": "state.clear_removes_active_preserves_history", + "observed": true, + "risk_id": "risk.clear_destroys_history", + "scenario_id": "real_manual_archive_and_leanoj_clear_preserve_scope_contract", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_build_d_proof_scope_filesystem.py" + }, + { + "adapter": "real_route", + "declared": true, + "field": "workflow_filesystem_state", + "invariant_id": "state.clear_removes_active_preserves_history", + "observed": true, + "risk_id": "risk.clear_destroys_history", + "scenario_id": "real_manual_compiler_papers_lifecycle_and_clear_archive", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" + }, + { + "adapter": "browser_smoke", + "declared": true, + "field": "websocket_api_contracts", + "invariant_id": "events.context_overflow_persists_across_reload", + "observed": true, + "risk_id": "risk.context_overflow_activity_loses_identity", + "scenario_id": "browser_overflow_activity_persists_with_attribution_and_terminal_deduplication", + "status": "passed", + "test_file": "tests/workflow_browser_smoke/overflow-activity.spec.js" + }, + { + "adapter": "model", + "declared": true, + "field": "websocket_api_contracts", + "invariant_id": "events.context_overflow_persists_across_reload", + "observed": true, + "risk_id": "risk.context_overflow_activity_loses_identity", + "scenario_id": "generated_context_overflow_contract_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "websocket_api_contracts", + "invariant_id": "events.context_overflow_persists_across_reload", + "observed": true, + "risk_id": "risk.context_overflow_activity_loses_identity", + "scenario_id": "generated_context_overflow_contract_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "real_coordinator", + "declared": true, + "field": "websocket_api_contracts", + "invariant_id": "events.context_overflow_persists_across_reload", + "observed": true, + "risk_id": "risk.context_overflow_activity_loses_identity", + "scenario_id": "real_manual_aggregator_overflow_identity_persists_across_reload", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_build_d_overflow_lifecycles.py" + }, + { + "adapter": "real_route", + "declared": false, + "field": "websocket_api_contracts", + "invariant_id": "events.context_overflow_persists_across_reload", + "observed": false, + "risk_id": "risk.context_overflow_activity_loses_identity", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "browser_smoke", + "declared": true, + "field": "websocket_api_contracts", + "invariant_id": "events.context_overflow_route_identity", + "observed": true, + "risk_id": "risk.context_overflow_activity_loses_identity", + "scenario_id": "browser_overflow_activity_persists_with_attribution_and_terminal_deduplication", + "status": "passed", + "test_file": "tests/workflow_browser_smoke/overflow-activity.spec.js" + }, + { + "adapter": "model", + "declared": true, + "field": "websocket_api_contracts", + "invariant_id": "events.context_overflow_route_identity", + "observed": true, + "risk_id": "risk.context_overflow_activity_loses_identity", + "scenario_id": "generated_context_overflow_contract_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "websocket_api_contracts", + "invariant_id": "events.context_overflow_route_identity", + "observed": true, + "risk_id": "risk.context_overflow_activity_loses_identity", + "scenario_id": "generated_context_overflow_contract_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "real_coordinator", + "declared": true, + "field": "websocket_api_contracts", + "invariant_id": "events.context_overflow_route_identity", + "observed": true, + "risk_id": "risk.context_overflow_activity_loses_identity", + "scenario_id": "real_autonomous_overflow_terminal_stop_once", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_build_d_overflow_lifecycles.py" + }, + { + "adapter": "real_coordinator", + "declared": true, + "field": "websocket_api_contracts", + "invariant_id": "events.context_overflow_route_identity", + "observed": true, + "risk_id": "risk.context_overflow_activity_loses_identity", + "scenario_id": "real_manual_aggregator_overflow_identity_persists_across_reload", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_build_d_overflow_lifecycles.py" + }, + { + "adapter": "real_coordinator", + "declared": true, + "field": "websocket_api_contracts", + "invariant_id": "events.context_overflow_route_identity", + "observed": true, + "risk_id": "risk.context_overflow_activity_loses_identity", + "scenario_id": "real_proof_context_overflow_is_scoped_nonfatal", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_build_d_overflow_lifecycles.py" + }, + { + "adapter": "real_route", + "declared": false, + "field": "websocket_api_contracts", + "invariant_id": "events.context_overflow_route_identity", + "observed": false, + "risk_id": "risk.context_overflow_activity_loses_identity", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "browser_smoke", + "declared": true, + "field": "websocket_api_contracts", + "invariant_id": "events.context_overflow_terminal_stop_once", + "observed": true, + "risk_id": "risk.context_overflow_activity_loses_identity", + "scenario_id": "browser_overflow_activity_persists_with_attribution_and_terminal_deduplication", + "status": "passed", + "test_file": "tests/workflow_browser_smoke/overflow-activity.spec.js" + }, + { + "adapter": "model", + "declared": true, + "field": "websocket_api_contracts", + "invariant_id": "events.context_overflow_terminal_stop_once", + "observed": true, + "risk_id": "risk.context_overflow_activity_loses_identity", + "scenario_id": "generated_context_overflow_contract_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "websocket_api_contracts", + "invariant_id": "events.context_overflow_terminal_stop_once", + "observed": true, + "risk_id": "risk.context_overflow_activity_loses_identity", + "scenario_id": "generated_context_overflow_contract_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "real_coordinator", + "declared": true, + "field": "websocket_api_contracts", + "invariant_id": "events.context_overflow_terminal_stop_once", + "observed": true, + "risk_id": "risk.context_overflow_activity_loses_identity", + "scenario_id": "real_autonomous_overflow_terminal_stop_once", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_build_d_overflow_lifecycles.py" + }, + { + "adapter": "real_route", + "declared": false, + "field": "websocket_api_contracts", + "invariant_id": "events.context_overflow_terminal_stop_once", + "observed": false, + "risk_id": "risk.context_overflow_activity_loses_identity", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "browser_smoke", + "declared": true, + "field": "websocket_api_contracts", + "invariant_id": "events.proof_context_overflow_nonfatal", + "observed": true, + "risk_id": "risk.context_overflow_activity_loses_identity", + "scenario_id": "browser_overflow_activity_persists_with_attribution_and_terminal_deduplication", + "status": "passed", + "test_file": "tests/workflow_browser_smoke/overflow-activity.spec.js" + }, + { + "adapter": "model", + "declared": true, + "field": "websocket_api_contracts", + "invariant_id": "events.proof_context_overflow_nonfatal", + "observed": true, + "risk_id": "risk.context_overflow_activity_loses_identity", + "scenario_id": "generated_context_overflow_contract_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "websocket_api_contracts", + "invariant_id": "events.proof_context_overflow_nonfatal", + "observed": true, + "risk_id": "risk.context_overflow_activity_loses_identity", + "scenario_id": "generated_context_overflow_contract_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "real_coordinator", + "declared": true, + "field": "websocket_api_contracts", + "invariant_id": "events.proof_context_overflow_nonfatal", + "observed": true, + "risk_id": "risk.context_overflow_activity_loses_identity", + "scenario_id": "real_proof_context_overflow_is_scoped_nonfatal", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_build_d_overflow_lifecycles.py" + }, + { + "adapter": "real_route", + "declared": false, + "field": "websocket_api_contracts", + "invariant_id": "events.proof_context_overflow_nonfatal", + "observed": false, + "risk_id": "risk.context_overflow_activity_loses_identity", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "browser_smoke", + "declared": false, + "field": "prompt_context", + "invariant_id": "prompt.direct_sources_excluded_from_rag", + "observed": false, + "risk_id": "risk.direct_source_duplicated_by_rag", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "model", + "declared": true, + "field": "prompt_context", + "invariant_id": "prompt.direct_sources_excluded_from_rag", + "observed": true, + "risk_id": "risk.direct_source_duplicated_by_rag", + "scenario_id": "generated_rag_prompt_boundaries_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "prompt_context", + "invariant_id": "prompt.direct_sources_excluded_from_rag", + "observed": true, + "risk_id": "risk.direct_source_duplicated_by_rag", + "scenario_id": "generated_rag_prompt_boundaries_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "real_coordinator", + "declared": true, + "field": "prompt_context", + "invariant_id": "prompt.direct_sources_excluded_from_rag", + "observed": true, + "risk_id": "risk.direct_source_duplicated_by_rag", + "scenario_id": "real_direct_source_rag_exclusion_matrix", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_build_e_rag_prompt_context.py" + }, + { + "adapter": "real_route", + "declared": false, + "field": "prompt_context", + "invariant_id": "prompt.direct_sources_excluded_from_rag", + "observed": false, + "risk_id": "risk.direct_source_duplicated_by_rag", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "browser_smoke", + "declared": false, + "field": "prompt_context", + "invariant_id": "prompt.mandatory_source_overflow_fails_visible", + "observed": false, + "risk_id": "risk.direct_source_duplicated_by_rag", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "model", + "declared": true, + "field": "prompt_context", + "invariant_id": "prompt.mandatory_source_overflow_fails_visible", + "observed": true, + "risk_id": "risk.direct_source_duplicated_by_rag", + "scenario_id": "generated_rag_prompt_boundaries_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "prompt_context", + "invariant_id": "prompt.mandatory_source_overflow_fails_visible", + "observed": true, + "risk_id": "risk.direct_source_duplicated_by_rag", + "scenario_id": "generated_rag_prompt_boundaries_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "real_coordinator", + "declared": true, + "field": "prompt_context", + "invariant_id": "prompt.mandatory_source_overflow_fails_visible", + "observed": true, + "risk_id": "risk.direct_source_duplicated_by_rag", + "scenario_id": "real_rigor_mandatory_source_overflow_fails_before_model_or_rag", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_build_e_prompt_boundaries.py" + }, + { + "adapter": "real_route", + "declared": false, + "field": "prompt_context", + "invariant_id": "prompt.mandatory_source_overflow_fails_visible", + "observed": false, + "risk_id": "risk.direct_source_duplicated_by_rag", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "browser_smoke", + "declared": false, + "field": "proof_runtime_gating", + "invariant_id": "proof_runtime.no_lean_when_disabled", + "observed": false, + "risk_id": "risk.disabled_proof_runtime_executes", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "model", + "declared": true, + "field": "proof_runtime_gating", + "invariant_id": "proof_runtime.no_lean_when_disabled", + "observed": true, + "risk_id": "risk.disabled_proof_runtime_executes", + "scenario_id": "generated_output_and_proof_runtime_gating_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "proof_runtime_gating", + "invariant_id": "proof_runtime.no_lean_when_disabled", + "observed": true, + "risk_id": "risk.disabled_proof_runtime_executes", + "scenario_id": "generated_output_and_proof_runtime_gating_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "proof_runtime_gating", + "invariant_id": "proof_runtime.no_lean_when_disabled", + "observed": true, + "risk_id": "risk.disabled_proof_runtime_executes", + "scenario_id": "model_proof_runtime_gating_allowed_outputs", + "status": "passed", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" + }, + { + "adapter": "real_coordinator", + "declared": true, + "field": "proof_runtime_gating", + "invariant_id": "proof_runtime.no_lean_when_disabled", + "observed": true, + "risk_id": "risk.disabled_proof_runtime_executes", + "scenario_id": "real_disabled_proof_runtime_skips_autonomous_checkpoint", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_high_value_scenarios.py" + }, + { + "adapter": "real_route", + "declared": false, + "field": "proof_runtime_gating", + "invariant_id": "proof_runtime.no_lean_when_disabled", + "observed": false, + "risk_id": "risk.disabled_proof_runtime_executes", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "browser_smoke", + "declared": false, + "field": "proof_runtime_gating", + "invariant_id": "proof_runtime.no_smt_when_disabled", + "observed": false, + "risk_id": "risk.disabled_proof_runtime_executes", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "model", + "declared": true, + "field": "proof_runtime_gating", + "invariant_id": "proof_runtime.no_smt_when_disabled", + "observed": true, + "risk_id": "risk.disabled_proof_runtime_executes", + "scenario_id": "generated_output_and_proof_runtime_gating_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "proof_runtime_gating", + "invariant_id": "proof_runtime.no_smt_when_disabled", + "observed": true, + "risk_id": "risk.disabled_proof_runtime_executes", + "scenario_id": "generated_output_and_proof_runtime_gating_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "proof_runtime_gating", + "invariant_id": "proof_runtime.no_smt_when_disabled", + "observed": true, + "risk_id": "risk.disabled_proof_runtime_executes", + "scenario_id": "model_proof_runtime_gating_allowed_outputs", + "status": "passed", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" + }, + { + "adapter": "real_coordinator", + "declared": false, + "field": "proof_runtime_gating", + "invariant_id": "proof_runtime.no_smt_when_disabled", + "observed": false, + "risk_id": "risk.disabled_proof_runtime_executes", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "real_route", + "declared": false, + "field": "proof_runtime_gating", + "invariant_id": "proof_runtime.no_smt_when_disabled", + "observed": false, + "risk_id": "risk.disabled_proof_runtime_executes", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "browser_smoke", + "declared": false, + "field": "prompt_context", + "invariant_id": "prompt.generated_appendices_stripped", + "observed": false, + "risk_id": "risk.generated_proofs_reenter_prompts", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "model", + "declared": true, + "field": "prompt_context", + "invariant_id": "prompt.generated_appendices_stripped", + "observed": true, + "risk_id": "risk.generated_proofs_reenter_prompts", + "scenario_id": "generated_rag_prompt_boundaries_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "prompt_context", + "invariant_id": "prompt.generated_appendices_stripped", + "observed": true, + "risk_id": "risk.generated_proofs_reenter_prompts", + "scenario_id": "generated_rag_prompt_boundaries_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "real_coordinator", + "declared": true, + "field": "prompt_context", + "invariant_id": "prompt.generated_appendices_stripped", + "observed": true, + "risk_id": "risk.generated_proofs_reenter_prompts", + "scenario_id": "real_model_visible_prompts_strip_generated_proof_appendices", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_build_e_prompt_boundaries.py" + }, + { + "adapter": "real_route", + "declared": false, + "field": "prompt_context", + "invariant_id": "prompt.generated_appendices_stripped", + "observed": false, + "risk_id": "risk.generated_proofs_reenter_prompts", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "browser_smoke", + "declared": false, + "field": "prompt_context", + "invariant_id": "prompt.proof_source_context_required", + "observed": false, + "risk_id": "risk.generated_proofs_reenter_prompts", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "model", + "declared": true, + "field": "prompt_context", + "invariant_id": "prompt.proof_source_context_required", + "observed": true, + "risk_id": "risk.generated_proofs_reenter_prompts", + "scenario_id": "generated_rag_prompt_boundaries_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "prompt_context", + "invariant_id": "prompt.proof_source_context_required", + "observed": true, + "risk_id": "risk.generated_proofs_reenter_prompts", + "scenario_id": "generated_rag_prompt_boundaries_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "real_coordinator", + "declared": false, + "field": "prompt_context", + "invariant_id": "prompt.proof_source_context_required", + "observed": false, + "risk_id": "risk.generated_proofs_reenter_prompts", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "real_route", + "declared": false, + "field": "prompt_context", + "invariant_id": "prompt.proof_source_context_required", + "observed": false, + "risk_id": "risk.generated_proofs_reenter_prompts", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "browser_smoke", + "declared": false, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.generated_appendices_stripped_from_prompts", + "observed": false, + "risk_id": "risk.generated_proofs_reenter_prompts", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "model", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.generated_appendices_stripped_from_prompts", + "observed": false, + "risk_id": "risk.generated_proofs_reenter_prompts", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "real_coordinator", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.generated_appendices_stripped_from_prompts", + "observed": true, + "risk_id": "risk.generated_proofs_reenter_prompts", + "scenario_id": "real_generated_proof_appendix_stripping", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" + }, + { + "adapter": "real_coordinator", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.generated_appendices_stripped_from_prompts", + "observed": true, + "risk_id": "risk.generated_proofs_reenter_prompts", + "scenario_id": "real_model_visible_prompts_strip_generated_proof_appendices", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_build_e_prompt_boundaries.py" + }, + { + "adapter": "real_route", + "declared": false, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.generated_appendices_stripped_from_prompts", + "observed": false, + "risk_id": "risk.generated_proofs_reenter_prompts", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "browser_smoke", + "declared": true, + "field": "websocket_api_contracts", + "invariant_id": "api.hosted_desktop_only_routes_unavailable", + "observed": true, + "risk_id": "risk.hosted_desktop_boundary", + "scenario_id": "browser_hosted_startup_respects_desktop_capability_boundaries", + "status": "passed", + "test_file": "tests/workflow_browser_smoke/hosted-capabilities.spec.js" + }, + { + "adapter": "model", + "declared": true, + "field": "websocket_api_contracts", + "invariant_id": "api.hosted_desktop_only_routes_unavailable", + "observed": false, + "risk_id": "risk.hosted_desktop_boundary", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "real_coordinator", + "declared": false, + "field": "websocket_api_contracts", + "invariant_id": "api.hosted_desktop_only_routes_unavailable", + "observed": false, + "risk_id": "risk.hosted_desktop_boundary", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "real_route", + "declared": false, + "field": "websocket_api_contracts", + "invariant_id": "api.hosted_desktop_only_routes_unavailable", + "observed": false, + "risk_id": "risk.hosted_desktop_boundary", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "browser_smoke", + "declared": false, + "field": "proof_runtime_gating", + "invariant_id": "proof_runtime.hosted_proof_settings_unavailable", + "observed": false, + "risk_id": "risk.hosted_desktop_boundary", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "model", + "declared": true, + "field": "proof_runtime_gating", + "invariant_id": "proof_runtime.hosted_proof_settings_unavailable", + "observed": false, + "risk_id": "risk.hosted_desktop_boundary", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "real_coordinator", + "declared": false, + "field": "proof_runtime_gating", + "invariant_id": "proof_runtime.hosted_proof_settings_unavailable", + "observed": false, + "risk_id": "risk.hosted_desktop_boundary", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "real_route", + "declared": true, + "field": "proof_runtime_gating", + "invariant_id": "proof_runtime.hosted_proof_settings_unavailable", + "observed": true, + "risk_id": "risk.hosted_desktop_boundary", + "scenario_id": "real_hosted_proof_settings_returns_desktop_unavailable", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_high_value_scenarios.py" + }, + { + "adapter": "browser_smoke", + "declared": false, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.autonomous_events_not_manual", + "observed": false, + "risk_id": "risk.proof_scope_leak", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "model", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.autonomous_events_not_manual", + "observed": true, + "risk_id": "risk.proof_scope_leak", + "scenario_id": "generated_scoped_registered_proof_events_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.autonomous_events_not_manual", + "observed": true, + "risk_id": "risk.proof_scope_leak", + "scenario_id": "generated_scoped_registered_proof_events_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.autonomous_events_not_manual", + "observed": true, + "risk_id": "risk.proof_scope_leak", + "scenario_id": "model_websocket_proof_scope_events", + "status": "passed", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" + }, + { + "adapter": "real_coordinator", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.autonomous_events_not_manual", + "observed": true, + "risk_id": "risk.proof_scope_leak", + "scenario_id": "real_leanoj_master_proof_stop_resume_preserves_isolated_state", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_proof_routes.py" + }, + { + "adapter": "real_route", + "declared": false, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.autonomous_events_not_manual", + "observed": false, + "risk_id": "risk.proof_scope_leak", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "browser_smoke", + "declared": false, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.manual_clear_archives_active", + "observed": false, + "risk_id": "risk.proof_scope_leak", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "model", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.manual_clear_archives_active", + "observed": true, + "risk_id": "risk.proof_scope_leak", + "scenario_id": "generated_manual_scope_clear_archive_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.manual_clear_archives_active", + "observed": true, + "risk_id": "risk.proof_scope_leak", + "scenario_id": "generated_manual_scope_clear_archive_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.manual_clear_archives_active", + "observed": true, + "risk_id": "risk.proof_scope_leak", + "scenario_id": "model_manual_proof_clear_scope_assistant", + "status": "passed", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" + }, + { + "adapter": "real_coordinator", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.manual_clear_archives_active", + "observed": true, + "risk_id": "risk.proof_scope_leak", + "scenario_id": "real_manual_archive_and_leanoj_clear_preserve_scope_contract", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_build_d_proof_scope_filesystem.py" + }, + { + "adapter": "real_route", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.manual_clear_archives_active", + "observed": true, + "risk_id": "risk.proof_scope_leak", + "scenario_id": "real_manual_aggregator_clear_archives_before_clear", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" + }, + { + "adapter": "real_route", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.manual_clear_archives_active", + "observed": true, + "risk_id": "risk.proof_scope_leak", + "scenario_id": "real_manual_compiler_papers_lifecycle_and_clear_archive", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" + }, + { + "adapter": "browser_smoke", + "declared": false, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.manual_not_in_autonomous_current", + "observed": false, + "risk_id": "risk.proof_scope_leak", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "model", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.manual_not_in_autonomous_current", + "observed": true, + "risk_id": "risk.proof_scope_leak", + "scenario_id": "generated_manual_scope_clear_archive_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.manual_not_in_autonomous_current", + "observed": true, + "risk_id": "risk.proof_scope_leak", + "scenario_id": "generated_manual_scope_clear_archive_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.manual_not_in_autonomous_current", + "observed": true, + "risk_id": "risk.proof_scope_leak", + "scenario_id": "model_manual_proof_clear_scope_assistant", + "status": "passed", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" + }, + { + "adapter": "real_coordinator", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.manual_not_in_autonomous_current", + "observed": true, + "risk_id": "risk.proof_scope_leak", + "scenario_id": "real_manual_compiler_proof_only_uses_rigor_settings_and_manual_scope", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_proof_routes.py" + }, + { + "adapter": "real_route", + "declared": true, + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.manual_not_in_autonomous_current", + "observed": true, + "risk_id": "risk.proof_scope_leak", + "scenario_id": "real_proof_scope_matrix_isolated_under_temp_root", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_build_d_proof_scope_filesystem.py" + }, + { + "adapter": "browser_smoke", + "declared": true, + "field": "allowed_outputs", + "invariant_id": "outputs.at_least_one_output_enabled", + "observed": true, + "risk_id": "risk.proofs_only_enters_paper", + "scenario_id": "browser_autonomous_start_stop_preserves_single_ui_owner", + "status": "passed", + "test_file": "tests/workflow_browser_smoke/autonomous-controls.spec.js" + }, + { + "adapter": "model", + "declared": true, + "field": "allowed_outputs", + "invariant_id": "outputs.at_least_one_output_enabled", + "observed": true, + "risk_id": "risk.proofs_only_enters_paper", + "scenario_id": "generated_output_and_proof_runtime_gating_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "allowed_outputs", + "invariant_id": "outputs.at_least_one_output_enabled", + "observed": true, + "risk_id": "risk.proofs_only_enters_paper", + "scenario_id": "generated_output_and_proof_runtime_gating_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "allowed_outputs", + "invariant_id": "outputs.at_least_one_output_enabled", + "observed": true, + "risk_id": "risk.proofs_only_enters_paper", + "scenario_id": "model_proof_runtime_gating_allowed_outputs", + "status": "passed", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" + }, + { + "adapter": "real_coordinator", + "declared": false, + "field": "allowed_outputs", + "invariant_id": "outputs.at_least_one_output_enabled", + "observed": false, + "risk_id": "risk.proofs_only_enters_paper", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "real_route", + "declared": true, + "field": "allowed_outputs", + "invariant_id": "outputs.at_least_one_output_enabled", + "observed": true, + "risk_id": "risk.proofs_only_enters_paper", + "scenario_id": "real_manual_compiler_rejects_no_allowed_outputs", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" + }, + { + "adapter": "browser_smoke", + "declared": false, + "field": "allowed_outputs", + "invariant_id": "outputs.proofs_only_no_paper_phase", + "observed": false, + "risk_id": "risk.proofs_only_enters_paper", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "model", + "declared": true, + "field": "allowed_outputs", + "invariant_id": "outputs.proofs_only_no_paper_phase", + "observed": true, + "risk_id": "risk.proofs_only_enters_paper", + "scenario_id": "generated_allowed_outputs_provider_pause_resume_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "allowed_outputs", + "invariant_id": "outputs.proofs_only_no_paper_phase", + "observed": true, + "risk_id": "risk.proofs_only_enters_paper", + "scenario_id": "generated_allowed_outputs_provider_pause_resume_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "allowed_outputs", + "invariant_id": "outputs.proofs_only_no_paper_phase", + "observed": true, + "risk_id": "risk.proofs_only_enters_paper", + "scenario_id": "model_allowed_outputs_provider_pause_stop_resume", + "status": "passed", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" + }, + { + "adapter": "real_coordinator", + "declared": true, + "field": "allowed_outputs", + "invariant_id": "outputs.proofs_only_no_paper_phase", + "observed": true, + "risk_id": "risk.proofs_only_enters_paper", + "scenario_id": "real_autonomous_proofs_only_handoff_returns_to_topic_exploration", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_proof_routes.py" + }, + { + "adapter": "real_route", + "declared": false, + "field": "allowed_outputs", + "invariant_id": "outputs.proofs_only_no_paper_phase", + "observed": false, + "risk_id": "risk.proofs_only_enters_paper", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "browser_smoke", + "declared": false, + "field": "provider_pause_resume", + "invariant_id": "provider.pause_preserves_checkpoint", + "observed": false, + "risk_id": "risk.provider_pause_loses_checkpoint", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "model", + "declared": true, + "field": "provider_pause_resume", + "invariant_id": "provider.pause_preserves_checkpoint", + "observed": true, + "risk_id": "risk.provider_pause_loses_checkpoint", + "scenario_id": "generated_allowed_outputs_provider_pause_resume_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "provider_pause_resume", + "invariant_id": "provider.pause_preserves_checkpoint", + "observed": true, + "risk_id": "risk.provider_pause_loses_checkpoint", + "scenario_id": "generated_allowed_outputs_provider_pause_resume_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "provider_pause_resume", + "invariant_id": "provider.pause_preserves_checkpoint", + "observed": true, + "risk_id": "risk.provider_pause_loses_checkpoint", + "scenario_id": "model_allowed_outputs_provider_pause_stop_resume", + "status": "passed", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" + }, + { + "adapter": "real_coordinator", + "declared": true, + "field": "provider_pause_resume", + "invariant_id": "provider.pause_preserves_checkpoint", + "observed": true, + "risk_id": "risk.provider_pause_loses_checkpoint", + "scenario_id": "real_autonomous_provider_credit_pause_preserves_checkpoint_and_resumes", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_proof_routes.py" + }, + { + "adapter": "real_route", + "declared": false, + "field": "provider_pause_resume", + "invariant_id": "provider.pause_preserves_checkpoint", + "observed": false, + "risk_id": "risk.provider_pause_loses_checkpoint", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "browser_smoke", + "declared": false, + "field": "provider_pause_resume", + "invariant_id": "provider.reset_wakes_without_corrupting_checkpoint", + "observed": false, + "risk_id": "risk.provider_pause_loses_checkpoint", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "model", + "declared": true, + "field": "provider_pause_resume", + "invariant_id": "provider.reset_wakes_without_corrupting_checkpoint", + "observed": true, + "risk_id": "risk.provider_pause_loses_checkpoint", + "scenario_id": "generated_allowed_outputs_provider_pause_resume_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "provider_pause_resume", + "invariant_id": "provider.reset_wakes_without_corrupting_checkpoint", + "observed": true, + "risk_id": "risk.provider_pause_loses_checkpoint", + "scenario_id": "generated_allowed_outputs_provider_pause_resume_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "provider_pause_resume", + "invariant_id": "provider.reset_wakes_without_corrupting_checkpoint", + "observed": true, + "risk_id": "risk.provider_pause_loses_checkpoint", + "scenario_id": "model_allowed_outputs_provider_pause_stop_resume", + "status": "passed", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" + }, + { + "adapter": "real_coordinator", + "declared": false, + "field": "provider_pause_resume", + "invariant_id": "provider.reset_wakes_without_corrupting_checkpoint", + "observed": false, + "risk_id": "risk.provider_pause_loses_checkpoint", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "real_route", + "declared": false, + "field": "provider_pause_resume", + "invariant_id": "provider.reset_wakes_without_corrupting_checkpoint", + "observed": false, + "risk_id": "risk.provider_pause_loses_checkpoint", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "browser_smoke", + "declared": false, + "field": "provider_pause_resume", + "invariant_id": "provider.stop_resume_preserves_pause", + "observed": false, + "risk_id": "risk.provider_pause_loses_checkpoint", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "model", + "declared": true, + "field": "provider_pause_resume", + "invariant_id": "provider.stop_resume_preserves_pause", + "observed": true, + "risk_id": "risk.provider_pause_loses_checkpoint", + "scenario_id": "generated_allowed_outputs_provider_pause_resume_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "provider_pause_resume", + "invariant_id": "provider.stop_resume_preserves_pause", + "observed": true, + "risk_id": "risk.provider_pause_loses_checkpoint", + "scenario_id": "generated_allowed_outputs_provider_pause_resume_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "provider_pause_resume", + "invariant_id": "provider.stop_resume_preserves_pause", + "observed": true, + "risk_id": "risk.provider_pause_loses_checkpoint", + "scenario_id": "generated_leanoj_durable_lifecycle_seed_29", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "provider_pause_resume", + "invariant_id": "provider.stop_resume_preserves_pause", + "observed": true, + "risk_id": "risk.provider_pause_loses_checkpoint", + "scenario_id": "generated_leanoj_durable_lifecycle_seed_7", + "status": "passed", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "declared": true, + "field": "provider_pause_resume", + "invariant_id": "provider.stop_resume_preserves_pause", + "observed": true, + "risk_id": "risk.provider_pause_loses_checkpoint", + "scenario_id": "model_allowed_outputs_provider_pause_stop_resume", + "status": "passed", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" + }, + { + "adapter": "real_coordinator", + "declared": false, + "field": "provider_pause_resume", + "invariant_id": "provider.stop_resume_preserves_pause", + "observed": true, + "reason": "Existing coordinator coverage observes pause/reset resume, but no bounded test seam independently interleaves user Stop while the provider wait is active.", + "risk_id": "risk.provider_pause_loses_checkpoint", + "scenario_id": "real_provider_stop_reset_checkpoint_unavailable_without_wait_seam", + "status": "blocked", + "test_file": "tests/workflow_real_adapters/test_high_value_scenarios.py" + }, + { + "adapter": "real_route", + "declared": false, + "field": "provider_pause_resume", + "invariant_id": "provider.stop_resume_preserves_pause", + "observed": false, + "risk_id": "risk.provider_pause_loses_checkpoint", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "browser_smoke", + "declared": false, + "field": "workflow_filesystem_state", + "invariant_id": "state.pruned_papers_excluded_from_context", + "observed": false, + "risk_id": "risk.pruned_paper_context_leak", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "model", + "declared": true, + "field": "workflow_filesystem_state", + "invariant_id": "state.pruned_papers_excluded_from_context", + "observed": false, + "risk_id": "risk.pruned_paper_context_leak", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "real_coordinator", + "declared": false, + "field": "workflow_filesystem_state", + "invariant_id": "state.pruned_papers_excluded_from_context", + "observed": false, + "risk_id": "risk.pruned_paper_context_leak", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "real_route", + "declared": true, + "field": "workflow_filesystem_state", + "invariant_id": "state.pruned_papers_excluded_from_context", + "observed": true, + "risk_id": "risk.pruned_paper_context_leak", + "scenario_id": "real_pruned_paper_preserved_but_removed_from_active_context", + "status": "passed", + "test_file": "tests/workflow_real_adapters/test_build_d_proof_scope_filesystem.py" + }, + { + "adapter": "browser_smoke", + "declared": false, + "field": "runtime_exclusivity", + "invariant_id": "runtime.parent_action_fences_child_outputs", + "observed": false, + "risk_id": "risk.stale_child_phase_takeover", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "model", + "declared": true, + "field": "runtime_exclusivity", + "invariant_id": "runtime.parent_action_fences_child_outputs", + "observed": false, + "risk_id": "risk.stale_child_phase_takeover", + "scenario_id": null, + "status": "uncovered", + "test_file": null + }, + { + "adapter": "real_coordinator", + "declared": false, + "field": "runtime_exclusivity", + "invariant_id": "runtime.parent_action_fences_child_outputs", + "observed": true, + "reason": "The invariant catalog declares model-only support and no existing isolated production seam exposes stale child output acceptance after parent takeover.", + "risk_id": "risk.stale_child_phase_takeover", + "scenario_id": "real_parent_action_fencing_unavailable_without_production_seam", + "status": "blocked", + "test_file": "tests/workflow_real_adapters/test_high_value_scenarios.py" + }, + { + "adapter": "real_route", + "declared": false, + "field": "runtime_exclusivity", + "invariant_id": "runtime.parent_action_fences_child_outputs", + "observed": false, + "risk_id": "risk.stale_child_phase_takeover", + "scenario_id": null, + "status": "uncovered", + "test_file": null + } + ], + "gaps": [ + { + "adapter": "real_coordinator", + "declared": false, + "field": "assistant_memory", + "gap_id": "risk.assistant_memory_crosses_boundary:assistant.disable_clears_live_pack:real_coordinator", + "invariant_id": "assistant.disable_clears_live_pack", + "reason": "No observed coverage record exists for this required risk/invariant/adapter basis.", + "risk_id": "risk.assistant_memory_crosses_boundary", + "status": "uncovered" + }, + { + "adapter": "real_coordinator", + "declared": false, + "field": "assistant_memory", + "gap_id": "risk.assistant_memory_crosses_boundary:assistant.no_validator_injection:real_coordinator", + "invariant_id": "assistant.no_validator_injection", + "reason": "No observed coverage record exists for this required risk/invariant/adapter basis.", + "risk_id": "risk.assistant_memory_crosses_boundary", + "status": "uncovered" + }, + { + "adapter": "real_coordinator", + "declared": false, + "field": "prompt_context", + "gap_id": "risk.assistant_memory_crosses_boundary:prompt.validator_excludes_assistant_memory:real_coordinator", + "invariant_id": "prompt.validator_excludes_assistant_memory", + "reason": "No observed coverage record exists for this required risk/invariant/adapter basis.", + "risk_id": "risk.assistant_memory_crosses_boundary", + "status": "uncovered" + }, + { + "adapter": "real_coordinator", + "declared": false, + "field": "proof_runtime_gating", + "gap_id": "risk.disabled_proof_runtime_executes:proof_runtime.no_smt_when_disabled:real_coordinator", + "invariant_id": "proof_runtime.no_smt_when_disabled", + "reason": "No observed coverage record exists for this required risk/invariant/adapter basis.", + "risk_id": "risk.disabled_proof_runtime_executes", + "status": "uncovered" + }, + { + "adapter": "real_coordinator", + "declared": false, + "field": "prompt_context", + "gap_id": "risk.generated_proofs_reenter_prompts:prompt.proof_source_context_required:real_coordinator", + "invariant_id": "prompt.proof_source_context_required", + "reason": "No observed coverage record exists for this required risk/invariant/adapter basis.", + "risk_id": "risk.generated_proofs_reenter_prompts", + "status": "uncovered" + }, + { + "adapter": "model", + "declared": true, + "field": "proof_scope_isolation", + "gap_id": "risk.generated_proofs_reenter_prompts:proof_scope.generated_appendices_stripped_from_prompts:model", + "invariant_id": "proof_scope.generated_appendices_stripped_from_prompts", + "reason": "No observed coverage record exists for this required risk/invariant/adapter basis.", + "risk_id": "risk.generated_proofs_reenter_prompts", + "status": "uncovered" + }, + { + "adapter": "model", + "declared": true, + "field": "websocket_api_contracts", + "gap_id": "risk.hosted_desktop_boundary:api.hosted_desktop_only_routes_unavailable:model", + "invariant_id": "api.hosted_desktop_only_routes_unavailable", + "reason": "No observed coverage record exists for this required risk/invariant/adapter basis.", + "risk_id": "risk.hosted_desktop_boundary", + "status": "uncovered" + }, + { + "adapter": "real_route", + "declared": false, + "field": "websocket_api_contracts", + "gap_id": "risk.hosted_desktop_boundary:api.hosted_desktop_only_routes_unavailable:real_route", + "invariant_id": "api.hosted_desktop_only_routes_unavailable", + "reason": "No observed coverage record exists for this required risk/invariant/adapter basis.", + "risk_id": "risk.hosted_desktop_boundary", + "status": "uncovered" + }, + { + "adapter": "model", + "declared": true, + "field": "proof_runtime_gating", + "gap_id": "risk.hosted_desktop_boundary:proof_runtime.hosted_proof_settings_unavailable:model", + "invariant_id": "proof_runtime.hosted_proof_settings_unavailable", + "reason": "No observed coverage record exists for this required risk/invariant/adapter basis.", + "risk_id": "risk.hosted_desktop_boundary", + "status": "uncovered" + }, + { + "adapter": "real_coordinator", + "declared": false, + "field": "allowed_outputs", + "gap_id": "risk.proofs_only_enters_paper:outputs.at_least_one_output_enabled:real_coordinator", + "invariant_id": "outputs.at_least_one_output_enabled", + "reason": "No observed coverage record exists for this required risk/invariant/adapter basis.", + "risk_id": "risk.proofs_only_enters_paper", + "status": "uncovered" + }, + { + "adapter": "real_coordinator", + "declared": false, + "field": "provider_pause_resume", + "gap_id": "risk.provider_pause_loses_checkpoint:provider.reset_wakes_without_corrupting_checkpoint:real_coordinator", + "invariant_id": "provider.reset_wakes_without_corrupting_checkpoint", + "reason": "No observed coverage record exists for this required risk/invariant/adapter basis.", + "risk_id": "risk.provider_pause_loses_checkpoint", + "status": "uncovered" + }, + { + "adapter": "model", + "declared": true, + "field": "workflow_filesystem_state", + "gap_id": "risk.pruned_paper_context_leak:state.pruned_papers_excluded_from_context:model", + "invariant_id": "state.pruned_papers_excluded_from_context", + "reason": "No observed coverage record exists for this required risk/invariant/adapter basis.", + "risk_id": "risk.pruned_paper_context_leak", + "status": "uncovered" + }, + { + "adapter": "real_coordinator", + "declared": false, + "field": "workflow_filesystem_state", + "gap_id": "risk.pruned_paper_context_leak:state.pruned_papers_excluded_from_context:real_coordinator", + "invariant_id": "state.pruned_papers_excluded_from_context", + "reason": "No observed coverage record exists for this required risk/invariant/adapter basis.", + "risk_id": "risk.pruned_paper_context_leak", + "status": "uncovered" + }, + { + "adapter": "model", + "declared": true, + "field": "runtime_exclusivity", + "gap_id": "risk.stale_child_phase_takeover:runtime.parent_action_fences_child_outputs:model", + "invariant_id": "runtime.parent_action_fences_child_outputs", + "reason": "No observed coverage record exists for this required risk/invariant/adapter basis.", + "risk_id": "risk.stale_child_phase_takeover", + "status": "uncovered" + } + ], + "nodes": { + "adapters": [ + "model", + "real_route", + "real_coordinator", + "browser_smoke" + ], + "invariants": [ + { + "declared_adapters": [ + "browser_smoke", + "model" + ], + "field": "websocket_api_contracts", + "invariant_id": "api.hosted_desktop_only_routes_unavailable" + }, + { + "declared_adapters": [ + "model" + ], + "field": "assistant_memory", + "invariant_id": "assistant.disable_clears_live_pack" + }, + { + "declared_adapters": [ + "model" + ], + "field": "assistant_memory", + "invariant_id": "assistant.no_validator_injection" + }, + { + "declared_adapters": [ + "model" + ], + "field": "assistant_memory", + "invariant_id": "assistant.non_blocking_retrieval" + }, + { + "declared_adapters": [ + "model" + ], + "field": "assistant_memory", + "invariant_id": "assistant.stagnant_pack_backoff_not_shutdown" + }, + { + "declared_adapters": [ + "browser_smoke", + "model", + "real_coordinator" + ], + "field": "websocket_api_contracts", + "invariant_id": "events.context_overflow_persists_across_reload" + }, + { + "declared_adapters": [ + "browser_smoke", + "model", + "real_coordinator" + ], + "field": "websocket_api_contracts", + "invariant_id": "events.context_overflow_route_identity" + }, + { + "declared_adapters": [ + "browser_smoke", + "model", + "real_coordinator" + ], + "field": "websocket_api_contracts", + "invariant_id": "events.context_overflow_terminal_stop_once" + }, + { + "declared_adapters": [ + "model" + ], + "field": "websocket_api_contracts", + "invariant_id": "events.frontend_events_include_scope_phase" + }, + { + "declared_adapters": [ + "browser_smoke", + "model", + "real_coordinator" + ], + "field": "websocket_api_contracts", + "invariant_id": "events.proof_context_overflow_nonfatal" + }, + { + "declared_adapters": [ + "model" + ], + "field": "websocket_api_contracts", + "invariant_id": "events.proof_verified_after_registration" + }, + { + "declared_adapters": [ + "browser_smoke", + "model", + "real_route" + ], + "field": "allowed_outputs", + "invariant_id": "outputs.at_least_one_output_enabled" + }, + { + "declared_adapters": [ + "model" + ], + "field": "allowed_outputs", + "invariant_id": "outputs.papers_only_skips_proof_work" + }, + { + "declared_adapters": [ + "model", + "real_coordinator" + ], + "field": "allowed_outputs", + "invariant_id": "outputs.proofs_only_no_paper_phase" + }, + { + "declared_adapters": [ + "model", + "real_coordinator" + ], + "field": "prompt_context", + "invariant_id": "prompt.direct_sources_excluded_from_rag" + }, + { + "declared_adapters": [ + "model", + "real_coordinator" + ], + "field": "prompt_context", + "invariant_id": "prompt.generated_appendices_stripped" + }, + { + "declared_adapters": [ + "model", + "real_coordinator" + ], + "field": "prompt_context", + "invariant_id": "prompt.mandatory_source_overflow_fails_visible" + }, + { + "declared_adapters": [ + "model" + ], + "field": "prompt_context", + "invariant_id": "prompt.proof_source_context_required" + }, + { + "declared_adapters": [ + "model" + ], + "field": "prompt_context", + "invariant_id": "prompt.user_prompt_direct_injected" + }, + { + "declared_adapters": [ + "model" + ], + "field": "prompt_context", + "invariant_id": "prompt.validator_excludes_assistant_memory" + }, + { + "declared_adapters": [ + "model", + "real_route" + ], + "field": "proof_runtime_gating", + "invariant_id": "proof_runtime.hosted_proof_settings_unavailable" + }, + { + "declared_adapters": [ + "model", + "real_coordinator" + ], + "field": "proof_runtime_gating", + "invariant_id": "proof_runtime.no_lean_when_disabled" + }, + { + "declared_adapters": [ + "model" + ], + "field": "proof_runtime_gating", + "invariant_id": "proof_runtime.no_smt_when_disabled" + }, + { + "declared_adapters": [ + "model" + ], + "field": "proof_runtime_gating", + "invariant_id": "proof_runtime.truncation_is_attempt_failure" + }, + { + "declared_adapters": [ + "model", + "real_coordinator" + ], + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.autonomous_events_not_manual" + }, + { + "declared_adapters": [ + "model", + "real_coordinator" + ], + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.generated_appendices_stripped_from_prompts" + }, + { + "declared_adapters": [ + "model", + "real_coordinator", + "real_route" + ], + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.manual_clear_archives_active" + }, + { + "declared_adapters": [ + "model", + "real_coordinator", + "real_route" + ], + "field": "proof_scope_isolation", + "invariant_id": "proof_scope.manual_not_in_autonomous_current" + }, + { + "declared_adapters": [ + "model", + "real_coordinator" + ], + "field": "provider_pause_resume", + "invariant_id": "provider.pause_preserves_checkpoint" + }, + { + "declared_adapters": [ + "model" + ], + "field": "provider_pause_resume", + "invariant_id": "provider.reset_wakes_without_corrupting_checkpoint" + }, + { + "declared_adapters": [ + "model" + ], + "field": "provider_pause_resume", + "invariant_id": "provider.stop_resume_preserves_pause" + }, + { + "declared_adapters": [ + "model", + "real_route" + ], + "field": "runtime_exclusivity", + "invariant_id": "runtime.child_tasks_count_as_active" + }, + { + "declared_adapters": [ + "model" + ], + "field": "runtime_exclusivity", + "invariant_id": "runtime.parent_action_fences_child_outputs" + }, + { + "declared_adapters": [ + "browser_smoke", + "model", + "real_route" + ], + "field": "runtime_exclusivity", + "invariant_id": "runtime.single_active_workflow" + }, + { + "declared_adapters": [ + "model", + "real_coordinator", + "real_route" + ], + "field": "workflow_filesystem_state", + "invariant_id": "state.clear_removes_active_preserves_history" + }, + { + "declared_adapters": [ + "model" + ], + "field": "workflow_filesystem_state", + "invariant_id": "state.completed_brainstorm_no_replay_handoff" + }, + { + "declared_adapters": [ + "model", + "real_route" + ], + "field": "workflow_filesystem_state", + "invariant_id": "state.pruned_papers_excluded_from_context" + }, + { + "declared_adapters": [ + "model", + "real_coordinator" + ], + "field": "workflow_filesystem_state", + "invariant_id": "state.runtime_roots_are_active_roots" + } + ], + "risks": [ + { + "priority": "high", + "risk_id": "risk.assistant_memory_crosses_boundary", + "title": "Assistant memory crosses role boundary" + }, + { + "priority": "high", + "risk_id": "risk.clear_destroys_history", + "title": "Clear destroys history or leaves active state" + }, + { + "priority": "critical", + "risk_id": "risk.context_overflow_activity_loses_identity", + "title": "Context overflow loses route identity or lifecycle semantics" + }, + { + "priority": "high", + "risk_id": "risk.direct_source_duplicated_by_rag", + "title": "Direct source is duplicated or displaced by RAG" + }, + { + "priority": "critical", + "risk_id": "risk.disabled_proof_runtime_executes", + "title": "Disabled proof runtime executes tools" + }, + { + "priority": "high", + "risk_id": "risk.generated_proofs_reenter_prompts", + "title": "Generated proofs reenter source prompts" + }, + { + "priority": "critical", + "risk_id": "risk.hosted_desktop_boundary", + "title": "Hosted mode invokes desktop-only behavior" + }, + { + "priority": "critical", + "risk_id": "risk.proof_scope_leak", + "title": "Proof records leak across scopes" + }, + { + "priority": "critical", + "risk_id": "risk.proofs_only_enters_paper", + "title": "Proofs-only run enters paper phases" + }, + { + "priority": "critical", + "risk_id": "risk.provider_pause_loses_checkpoint", + "title": "Provider pause loses resumable work" + }, + { + "priority": "high", + "risk_id": "risk.pruned_paper_context_leak", + "title": "Pruned paper remains in model context" + }, + { + "priority": "critical", + "risk_id": "risk.stale_child_phase_takeover", + "title": "Stale child output overrides parent phase" + } + ], + "scenarios": [ + { + "adapter": "browser_smoke", + "result": "passed", + "scenario_id": "browser_autonomous_start_stop_preserves_single_ui_owner", + "test_file": "tests/workflow_browser_smoke/autonomous-controls.spec.js" + }, + { + "adapter": "browser_smoke", + "result": "passed", + "scenario_id": "browser_hosted_startup_respects_desktop_capability_boundaries", + "test_file": "tests/workflow_browser_smoke/hosted-capabilities.spec.js" + }, + { + "adapter": "browser_smoke", + "result": "passed", + "scenario_id": "browser_mode_switching_preserves_in_memory_default_and_developer_gate", + "test_file": "tests/workflow_browser_smoke/mode-switching.spec.js" + }, + { + "adapter": "browser_smoke", + "result": "passed", + "scenario_id": "browser_overflow_activity_persists_with_attribution_and_terminal_deduplication", + "test_file": "tests/workflow_browser_smoke/overflow-activity.spec.js" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "generated_allowed_outputs_provider_pause_resume_seed_29", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "generated_allowed_outputs_provider_pause_resume_seed_7", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "generated_assistant_prompt_context_seed_29", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "generated_assistant_prompt_context_seed_7", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "generated_autonomous_paper_checkpoint_seed_29", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "generated_autonomous_paper_checkpoint_seed_7", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "generated_context_overflow_contract_seed_29", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "generated_context_overflow_contract_seed_7", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "generated_leanoj_durable_lifecycle_seed_29", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "generated_leanoj_durable_lifecycle_seed_7", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "generated_manual_aggregator_lifecycle_seed_29", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "generated_manual_aggregator_lifecycle_seed_7", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "generated_manual_scope_clear_archive_seed_29", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "generated_manual_scope_clear_archive_seed_7", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "generated_output_and_proof_runtime_gating_seed_29", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "generated_output_and_proof_runtime_gating_seed_7", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "generated_rag_prompt_boundaries_seed_29", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "generated_rag_prompt_boundaries_seed_7", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "generated_runtime_start_exclusivity_seed_29", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "generated_runtime_start_exclusivity_seed_7", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "generated_scoped_registered_proof_events_seed_29", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "generated_scoped_registered_proof_events_seed_7", + "test_file": "tests/workflow_recombinatory/test_mode_aware_generator.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "model_allowed_outputs_provider_pause_stop_resume", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "model_assistant_prompt_validator_exclusion", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "model_manual_proof_clear_scope_assistant", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "model_proof_runtime_gating_allowed_outputs", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" + }, + { + "adapter": "model", + "result": "passed", + "scenario_id": "model_websocket_proof_scope_events", + "test_file": "tests/workflow_scenarios/test_cross_field_analysis.py" + }, + { + "adapter": "real_route", + "result": "passed", + "scenario_id": "real_actual_route_start_conflict_matrix_no_side_effects", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" + }, + { + "adapter": "real_coordinator", + "result": "passed", + "scenario_id": "real_autonomous_overflow_terminal_stop_once", + "test_file": "tests/workflow_real_adapters/test_build_d_overflow_lifecycles.py" + }, + { + "adapter": "real_coordinator", + "result": "passed", + "scenario_id": "real_autonomous_proofs_only_handoff_returns_to_topic_exploration", + "test_file": "tests/workflow_real_adapters/test_proof_routes.py" + }, + { + "adapter": "real_coordinator", + "result": "passed", + "scenario_id": "real_autonomous_provider_credit_pause_preserves_checkpoint_and_resumes", + "test_file": "tests/workflow_real_adapters/test_proof_routes.py" + }, + { + "adapter": "real_coordinator", + "result": "passed", + "scenario_id": "real_direct_source_rag_exclusion_matrix", + "test_file": "tests/workflow_real_adapters/test_build_e_rag_prompt_context.py" + }, + { + "adapter": "real_coordinator", + "result": "passed", + "scenario_id": "real_disabled_proof_runtime_skips_autonomous_checkpoint", + "test_file": "tests/workflow_real_adapters/test_high_value_scenarios.py" + }, + { + "adapter": "real_coordinator", + "result": "passed", + "scenario_id": "real_generated_proof_appendix_stripping", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" + }, + { + "adapter": "real_route", + "result": "passed", + "scenario_id": "real_hosted_proof_settings_returns_desktop_unavailable", + "test_file": "tests/workflow_real_adapters/test_high_value_scenarios.py" + }, + { + "adapter": "real_coordinator", + "result": "passed", + "scenario_id": "real_leanoj_coordinator_skip_force_clear_and_intermediate_edit", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" + }, + { + "adapter": "real_coordinator", + "result": "blocked", + "scenario_id": "real_leanoj_full_final_loop_not_safely_bounded", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" + }, + { + "adapter": "real_coordinator", + "result": "passed", + "scenario_id": "real_leanoj_master_proof_stop_resume_preserves_isolated_state", + "test_file": "tests/workflow_real_adapters/test_proof_routes.py" + }, + { + "adapter": "real_route", + "result": "passed", + "scenario_id": "real_manual_aggregator_clear_archives_before_clear", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" + }, + { + "adapter": "real_coordinator", + "result": "passed", + "scenario_id": "real_manual_aggregator_durable_hydration_and_clear_blocker", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" + }, + { + "adapter": "real_coordinator", + "result": "passed", + "scenario_id": "real_manual_aggregator_overflow_identity_persists_across_reload", + "test_file": "tests/workflow_real_adapters/test_build_d_overflow_lifecycles.py" + }, + { + "adapter": "real_route", + "result": "passed", + "scenario_id": "real_manual_aggregator_route_lifecycle_output_and_temp_root", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" + }, + { + "adapter": "real_coordinator", + "result": "passed", + "scenario_id": "real_manual_archive_and_leanoj_clear_preserve_scope_contract", + "test_file": "tests/workflow_real_adapters/test_build_d_proof_scope_filesystem.py" + }, + { + "adapter": "real_route", + "result": "passed", + "scenario_id": "real_manual_compiler_papers_lifecycle_and_clear_archive", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" + }, + { + "adapter": "real_route", + "result": "passed", + "scenario_id": "real_manual_compiler_proof_only_background_ownership", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" + }, + { + "adapter": "real_coordinator", + "result": "passed", + "scenario_id": "real_manual_compiler_proof_only_uses_rigor_settings_and_manual_scope", + "test_file": "tests/workflow_real_adapters/test_proof_routes.py" + }, + { + "adapter": "real_route", + "result": "passed", + "scenario_id": "real_manual_compiler_rejects_no_allowed_outputs", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" + }, + { + "adapter": "real_coordinator", + "result": "passed", + "scenario_id": "real_model_visible_prompts_strip_generated_proof_appendices", + "test_file": "tests/workflow_real_adapters/test_build_e_prompt_boundaries.py" + }, + { + "adapter": "real_coordinator", + "result": "blocked", + "scenario_id": "real_parent_action_fencing_unavailable_without_production_seam", + "test_file": "tests/workflow_real_adapters/test_high_value_scenarios.py" + }, + { + "adapter": "real_coordinator", + "result": "passed", + "scenario_id": "real_proof_context_overflow_is_scoped_nonfatal", + "test_file": "tests/workflow_real_adapters/test_build_d_overflow_lifecycles.py" + }, + { + "adapter": "real_route", + "result": "passed", + "scenario_id": "real_proof_scope_matrix_isolated_under_temp_root", + "test_file": "tests/workflow_real_adapters/test_build_d_proof_scope_filesystem.py" + }, + { + "adapter": "real_coordinator", + "result": "blocked", + "scenario_id": "real_provider_stop_reset_checkpoint_unavailable_without_wait_seam", + "test_file": "tests/workflow_real_adapters/test_high_value_scenarios.py" + }, + { + "adapter": "real_route", + "result": "passed", + "scenario_id": "real_pruned_paper_preserved_but_removed_from_active_context", + "test_file": "tests/workflow_real_adapters/test_build_d_proof_scope_filesystem.py" + }, + { + "adapter": "real_coordinator", + "result": "passed", + "scenario_id": "real_rigor_mandatory_source_overflow_fails_before_model_or_rag", + "test_file": "tests/workflow_real_adapters/test_build_e_prompt_boundaries.py" + }, + { + "adapter": "real_route", + "result": "passed", + "scenario_id": "real_route_pending_child_activity_counts_as_active", + "test_file": "tests/workflow_real_adapters/test_route_conflicts.py" + }, + { + "adapter": "real_route", + "result": "passed", + "scenario_id": "real_route_start_conflicts_preserve_single_workflow", + "test_file": "tests/workflow_real_adapters/test_route_conflicts.py" + }, + { + "adapter": "real_route", + "result": "passed", + "scenario_id": "real_shared_start_guard_representative_race", + "test_file": "tests/workflow_real_adapters/test_build_bc_lifecycles.py" + } + ] + }, + "schema_version": 1 +} diff --git a/tests/workflow_cross_field/support_graph.py b/tests/workflow_cross_field/support_graph.py new file mode 100644 index 0000000..02881e6 --- /dev/null +++ b/tests/workflow_cross_field/support_graph.py @@ -0,0 +1,250 @@ +from __future__ import annotations + +import json +from dataclasses import dataclass +from pathlib import Path +from typing import Iterable, Literal + +from tests.workflow_cross_field.target_risks import TARGET_RISKS +from tests.workflow_harness.coverage_metadata import InteractionCoverage +from tests.workflow_harness.invariant_catalog import INVARIANTS_BY_ID + + +SUPPORT_GRAPH_SCHEMA_VERSION = 1 +SUPPORT_GRAPH_JSON_PATH = Path(__file__).parent / "support_graph.json" +SUPPORT_GRAPH_MARKDOWN_PATH = Path(__file__).parent / "SUPPORT_GRAPH.md" + +SupportStatus = Literal["passed", "failed", "skipped", "blocked", "uncovered"] +ADAPTERS = ("model", "real_route", "real_coordinator", "browser_smoke") +STATUS_ORDER = {"failed": 0, "blocked": 1, "skipped": 2, "passed": 3, "uncovered": 4} + + +@dataclass(frozen=True) +class SupportEdge: + risk_id: str + invariant_id: str + field: str + scenario_id: str | None + adapter: str + status: SupportStatus + declared: bool + observed: bool + test_file: str | None = None + reason: str | None = None + + +def build_support_graph(records: Iterable[InteractionCoverage]) -> dict[str, object]: + ordered_records = tuple(sorted(records, key=_record_key)) + edges: list[SupportEdge] = [] + gaps: list[dict[str, object]] = [] + + for risk in sorted(TARGET_RISKS, key=lambda item: item.risk_id): + for invariant_id in sorted(risk.supporting_invariants): + invariant = INVARIANTS_BY_ID[invariant_id] + for adapter in ADAPTERS: + matches = tuple( + record + for record in ordered_records + if record.adapter == adapter and invariant_id in record.invariants + ) + declared = adapter in invariant.adapters + if matches: + for record in matches: + reason = _reason(record) + edges.append( + SupportEdge( + risk_id=risk.risk_id, + invariant_id=invariant_id, + field=invariant.field, + scenario_id=record.scenario_id, + adapter=adapter, + status=record.result, + declared=declared, + observed=True, + test_file=record.test_file, + reason=reason, + ) + ) + else: + edges.append( + SupportEdge( + risk_id=risk.risk_id, + invariant_id=invariant_id, + field=invariant.field, + scenario_id=None, + adapter=adapter, + status="uncovered", + declared=declared, + observed=False, + ) + ) + if adapter in risk.required_adapters: + gaps.append( + { + "gap_id": f"{risk.risk_id}:{invariant_id}:{adapter}", + "risk_id": risk.risk_id, + "invariant_id": invariant_id, + "field": invariant.field, + "adapter": adapter, + "status": "uncovered", + "declared": declared, + "reason": ( + "No observed coverage record exists for this required " + "risk/invariant/adapter basis." + ), + } + ) + + edge_payloads = [_edge_payload(edge) for edge in sorted(edges, key=_edge_key)] + scenario_nodes = sorted( + { + record.scenario_id: { + "scenario_id": record.scenario_id, + "adapter": record.adapter, + "result": record.result, + "test_file": record.test_file, + } + for record in ordered_records + }.values(), + key=lambda item: item["scenario_id"], + ) + return { + "schema_version": SUPPORT_GRAPH_SCHEMA_VERSION, + "artifact_type": "support_graph", + "basis": { + "declared": "Invariant catalog adapter declarations and target-risk requirements.", + "observed": "InteractionCoverage records produced by executable tests or truthful gap records.", + }, + "nodes": { + "risks": [ + {"risk_id": risk.risk_id, "title": risk.title, "priority": risk.priority} + for risk in sorted(TARGET_RISKS, key=lambda item: item.risk_id) + ], + "invariants": [ + { + "invariant_id": invariant_id, + "field": spec.field, + "declared_adapters": sorted(spec.adapters), + } + for invariant_id, spec in sorted(INVARIANTS_BY_ID.items()) + ], + "scenarios": scenario_nodes, + "adapters": list(ADAPTERS), + }, + "edges": edge_payloads, + "gaps": sorted(gaps, key=lambda item: item["gap_id"]), + } + + +def validate_support_graph(graph: dict[str, object]) -> None: + if graph.get("schema_version") != SUPPORT_GRAPH_SCHEMA_VERSION: + raise AssertionError("Unexpected support graph schema version.") + edges = graph.get("edges") + gaps = graph.get("gaps") + if not isinstance(edges, list) or not edges: + raise AssertionError("Support graph must contain edges.") + if not isinstance(gaps, list): + raise AssertionError("Support graph gaps must be a list.") + edge_keys: set[tuple[object, ...]] = set() + for edge in edges: + if not isinstance(edge, dict): + raise AssertionError("Support graph edge must be an object.") + required = {"risk_id", "invariant_id", "field", "adapter", "status", "declared", "observed"} + if not required.issubset(edge): + raise AssertionError(f"Support graph edge is missing fields: {required - set(edge)}") + if edge["status"] not in STATUS_ORDER: + raise AssertionError(f"Unknown support status {edge['status']!r}.") + if edge["observed"] and not edge.get("scenario_id"): + raise AssertionError("Observed support edge must identify its scenario.") + if not edge["observed"] and edge["status"] != "uncovered": + raise AssertionError("Unobserved support edge must be uncovered.") + key = tuple(edge.get(name) for name in ("risk_id", "invariant_id", "adapter", "scenario_id")) + if key in edge_keys: + raise AssertionError(f"Duplicate support edge {key!r}.") + edge_keys.add(key) + gap_ids = [gap.get("gap_id") for gap in gaps if isinstance(gap, dict)] + if len(gap_ids) != len(set(gap_ids)): + raise AssertionError("Support graph gap ids must be unique.") + + +def render_support_graph_json(records: Iterable[InteractionCoverage]) -> bytes: + graph = build_support_graph(records) + validate_support_graph(graph) + return (json.dumps(graph, indent=2, sort_keys=True) + "\n").encode() + + +def render_support_graph_markdown(records: Iterable[InteractionCoverage]) -> bytes: + graph = build_support_graph(records) + validate_support_graph(graph) + lines = [ + "# Workflow Support Graph", + "", + "Generated deterministically from separately versioned declared and observed support bases.", + "", + f"Schema version: {SUPPORT_GRAPH_SCHEMA_VERSION}.", + "", + "| Risk | Invariant | Field | Adapter | Status | Basis | Scenario / reason |", + "|---|---|---|---|---|---|---|", + ] + for edge in graph["edges"]: + basis = f"declared={'yes' if edge['declared'] else 'no'}, observed={'yes' if edge['observed'] else 'no'}" + detail = edge.get("scenario_id") or edge.get("reason") or "no observed scenario" + lines.append( + f"| `{edge['risk_id']}` | `{edge['invariant_id']}` | `{edge['field']}` | " + f"`{edge['adapter']}` | {edge['status']} | {basis} | {detail} |" + ) + lines.extend( + [ + "", + "## Structured Gaps", + "", + "| Gap | Status | Declared | Reason |", + "|---|---|---|---|", + ] + ) + for gap in graph["gaps"]: + lines.append( + f"| `{gap['gap_id']}` | {gap['status']} | " + f"{'yes' if gap['declared'] else 'no'} | {gap['reason']} |" + ) + if not graph["gaps"]: + lines.append("| none | passed | n/a | All required bases are observed. |") + return ("\n".join(lines) + "\n").encode() + + +def _reason(record: InteractionCoverage) -> str | None: + if record.result not in {"blocked", "skipped", "failed"} or not record.diagnostics: + return None + return record.diagnostics.get("reason") + + +def _record_key(record: InteractionCoverage) -> tuple[object, ...]: + return record.scenario_id, record.adapter, record.result, record.test_file + + +def _edge_key(edge: SupportEdge) -> tuple[object, ...]: + return ( + edge.risk_id, + edge.invariant_id, + edge.field, + edge.adapter, + STATUS_ORDER[edge.status], + edge.scenario_id or "", + ) + + +def _edge_payload(edge: SupportEdge) -> dict[str, object]: + payload: dict[str, object] = { + "risk_id": edge.risk_id, + "invariant_id": edge.invariant_id, + "field": edge.field, + "scenario_id": edge.scenario_id, + "adapter": edge.adapter, + "status": edge.status, + "declared": edge.declared, + "observed": edge.observed, + "test_file": edge.test_file, + } + if edge.reason: + payload["reason"] = edge.reason + return payload diff --git a/tests/workflow_cross_field/target_risks.json b/tests/workflow_cross_field/target_risks.json new file mode 100644 index 0000000..63f4865 --- /dev/null +++ b/tests/workflow_cross_field/target_risks.json @@ -0,0 +1,698 @@ +{ + "artifact_type": "target_risk_analysis", + "risks": [ + { + "description": "Hosted mode reaches proof settings or another desktop-only route or behavior.", + "existing_scenarios": [ + "real_hosted_proof_settings_returns_desktop_unavailable" + ], + "gap_score": 15, + "missing_isolated_artifacts": [ + "model:api.hosted_desktop_only_routes_unavailable", + "model:proof_runtime.hosted_proof_settings_unavailable", + "real_route:api.hosted_desktop_only_routes_unavailable" + ], + "priority": "critical", + "rank": 1, + "required_adapters": [ + "model", + "real_route" + ], + "required_fields": [ + "proof_runtime_gating", + "websocket_api_contracts", + "runtime_exclusivity", + "allowed_outputs" + ], + "risk_id": "risk.hosted_desktop_boundary", + "support": [ + { + "adapter": "model", + "missing_invariants": [ + "api.hosted_desktop_only_routes_unavailable", + "proof_runtime.hosted_proof_settings_unavailable" + ], + "result_statuses": [], + "scenario_ids": [], + "status": "uncovered" + }, + { + "adapter": "real_route", + "missing_invariants": [ + "api.hosted_desktop_only_routes_unavailable" + ], + "result_statuses": [ + "passed" + ], + "scenario_ids": [ + "real_hosted_proof_settings_returns_desktop_unavailable" + ], + "status": "uncovered" + } + ], + "supporting_invariants": [ + "proof_runtime.hosted_proof_settings_unavailable", + "api.hosted_desktop_only_routes_unavailable" + ], + "title": "Hosted mode invokes desktop-only behavior" + }, + { + "description": "Late child output changes state after a parent or user action takes ownership.", + "existing_scenarios": [ + "real_parent_action_fencing_unavailable_without_production_seam" + ], + "gap_score": 12, + "missing_isolated_artifacts": [ + "model:runtime.parent_action_fences_child_outputs", + "real_coordinator:runtime.parent_action_fences_child_outputs" + ], + "priority": "critical", + "rank": 2, + "required_adapters": [ + "model", + "real_coordinator" + ], + "required_fields": [ + "runtime_exclusivity", + "workflow_filesystem_state", + "websocket_api_contracts" + ], + "risk_id": "risk.stale_child_phase_takeover", + "support": [ + { + "adapter": "model", + "missing_invariants": [ + "runtime.parent_action_fences_child_outputs" + ], + "result_statuses": [], + "scenario_ids": [], + "status": "uncovered" + }, + { + "adapter": "real_coordinator", + "missing_invariants": [ + "runtime.parent_action_fences_child_outputs" + ], + "result_statuses": [ + "blocked" + ], + "scenario_ids": [ + "real_parent_action_fencing_unavailable_without_production_seam" + ], + "status": "blocked" + } + ], + "supporting_invariants": [ + "runtime.parent_action_fences_child_outputs" + ], + "title": "Stale child output overrides parent phase" + }, + { + "description": "Credit pause, stop, or reset loses or corrupts the durable proof cursor.", + "existing_scenarios": [ + "generated_allowed_outputs_provider_pause_resume_seed_29", + "generated_allowed_outputs_provider_pause_resume_seed_7", + "generated_leanoj_durable_lifecycle_seed_29", + "generated_leanoj_durable_lifecycle_seed_7", + "model_allowed_outputs_provider_pause_stop_resume", + "real_autonomous_provider_credit_pause_preserves_checkpoint_and_resumes", + "real_provider_stop_reset_checkpoint_unavailable_without_wait_seam" + ], + "gap_score": 9, + "missing_isolated_artifacts": [ + "real_coordinator:provider.reset_wakes_without_corrupting_checkpoint", + "real_coordinator:provider.stop_resume_preserves_pause" + ], + "priority": "critical", + "rank": 3, + "required_adapters": [ + "model", + "real_coordinator" + ], + "required_fields": [ + "provider_pause_resume", + "workflow_filesystem_state", + "proof_runtime_gating" + ], + "risk_id": "risk.provider_pause_loses_checkpoint", + "support": [ + { + "adapter": "model", + "missing_invariants": [], + "result_statuses": [ + "passed" + ], + "scenario_ids": [ + "generated_allowed_outputs_provider_pause_resume_seed_29", + "generated_allowed_outputs_provider_pause_resume_seed_7", + "generated_leanoj_durable_lifecycle_seed_29", + "generated_leanoj_durable_lifecycle_seed_7", + "model_allowed_outputs_provider_pause_stop_resume" + ], + "status": "passed" + }, + { + "adapter": "real_coordinator", + "missing_invariants": [ + "provider.reset_wakes_without_corrupting_checkpoint", + "provider.stop_resume_preserves_pause" + ], + "result_statuses": [ + "blocked", + "passed" + ], + "scenario_ids": [ + "real_autonomous_provider_credit_pause_preserves_checkpoint_and_resumes", + "real_provider_stop_reset_checkpoint_unavailable_without_wait_seam" + ], + "status": "blocked" + } + ], + "supporting_invariants": [ + "provider.pause_preserves_checkpoint", + "provider.stop_resume_preserves_pause", + "provider.reset_wakes_without_corrupting_checkpoint" + ], + "title": "Provider pause loses resumable work" + }, + { + "description": "Assistant proof memory reaches validators or remains live after disable.", + "existing_scenarios": [ + "generated_assistant_prompt_context_seed_29", + "generated_assistant_prompt_context_seed_7", + "model_assistant_prompt_validator_exclusion", + "model_manual_proof_clear_scope_assistant" + ], + "gap_score": 8, + "missing_isolated_artifacts": [ + "real_coordinator:assistant.disable_clears_live_pack", + "real_coordinator:assistant.no_validator_injection", + "real_coordinator:prompt.validator_excludes_assistant_memory" + ], + "priority": "high", + "rank": 4, + "required_adapters": [ + "model", + "real_coordinator" + ], + "required_fields": [ + "assistant_memory", + "prompt_context" + ], + "risk_id": "risk.assistant_memory_crosses_boundary", + "support": [ + { + "adapter": "model", + "missing_invariants": [], + "result_statuses": [ + "passed" + ], + "scenario_ids": [ + "generated_assistant_prompt_context_seed_29", + "generated_assistant_prompt_context_seed_7", + "model_assistant_prompt_validator_exclusion", + "model_manual_proof_clear_scope_assistant" + ], + "status": "passed" + }, + { + "adapter": "real_coordinator", + "missing_invariants": [ + "assistant.disable_clears_live_pack", + "assistant.no_validator_injection", + "prompt.validator_excludes_assistant_memory" + ], + "result_statuses": [], + "scenario_ids": [], + "status": "uncovered" + } + ], + "supporting_invariants": [ + "assistant.no_validator_injection", + "assistant.disable_clears_live_pack", + "prompt.validator_excludes_assistant_memory" + ], + "title": "Assistant memory crosses role boundary" + }, + { + "description": "Generated appendices duplicate into source context or replace mandatory source text.", + "existing_scenarios": [ + "generated_rag_prompt_boundaries_seed_29", + "generated_rag_prompt_boundaries_seed_7", + "real_generated_proof_appendix_stripping", + "real_model_visible_prompts_strip_generated_proof_appendices" + ], + "gap_score": 8, + "missing_isolated_artifacts": [ + "model:proof_scope.generated_appendices_stripped_from_prompts", + "real_coordinator:prompt.proof_source_context_required" + ], + "priority": "high", + "rank": 5, + "required_adapters": [ + "model", + "real_coordinator" + ], + "required_fields": [ + "prompt_context", + "proof_scope_isolation", + "workflow_filesystem_state" + ], + "risk_id": "risk.generated_proofs_reenter_prompts", + "support": [ + { + "adapter": "model", + "missing_invariants": [ + "proof_scope.generated_appendices_stripped_from_prompts" + ], + "result_statuses": [ + "passed" + ], + "scenario_ids": [ + "generated_rag_prompt_boundaries_seed_29", + "generated_rag_prompt_boundaries_seed_7" + ], + "status": "uncovered" + }, + { + "adapter": "real_coordinator", + "missing_invariants": [ + "prompt.proof_source_context_required" + ], + "result_statuses": [ + "passed" + ], + "scenario_ids": [ + "real_generated_proof_appendix_stripping", + "real_model_visible_prompts_strip_generated_proof_appendices" + ], + "status": "uncovered" + } + ], + "supporting_invariants": [ + "prompt.generated_appendices_stripped", + "proof_scope.generated_appendices_stripped_from_prompts", + "prompt.proof_source_context_required" + ], + "title": "Generated proofs reenter source prompts" + }, + { + "description": "A pruned paper remains available to model or RAG context after removal.", + "existing_scenarios": [], + "gap_score": 8, + "missing_isolated_artifacts": [ + "model:state.pruned_papers_excluded_from_context", + "real_coordinator:state.pruned_papers_excluded_from_context" + ], + "priority": "high", + "rank": 6, + "required_adapters": [ + "model", + "real_coordinator" + ], + "required_fields": [ + "workflow_filesystem_state", + "prompt_context" + ], + "risk_id": "risk.pruned_paper_context_leak", + "support": [ + { + "adapter": "model", + "missing_invariants": [ + "state.pruned_papers_excluded_from_context" + ], + "result_statuses": [], + "scenario_ids": [], + "status": "uncovered" + }, + { + "adapter": "real_coordinator", + "missing_invariants": [ + "state.pruned_papers_excluded_from_context" + ], + "result_statuses": [], + "scenario_ids": [], + "status": "uncovered" + } + ], + "supporting_invariants": [ + "state.pruned_papers_excluded_from_context" + ], + "title": "Pruned paper remains in model context" + }, + { + "description": "Lean or SMT runs despite disabled proof-runtime controls.", + "existing_scenarios": [ + "generated_output_and_proof_runtime_gating_seed_29", + "generated_output_and_proof_runtime_gating_seed_7", + "model_proof_runtime_gating_allowed_outputs", + "real_disabled_proof_runtime_skips_autonomous_checkpoint" + ], + "gap_score": 6, + "missing_isolated_artifacts": [ + "real_coordinator:proof_runtime.no_smt_when_disabled" + ], + "priority": "critical", + "rank": 7, + "required_adapters": [ + "model", + "real_coordinator" + ], + "required_fields": [ + "proof_runtime_gating", + "allowed_outputs" + ], + "risk_id": "risk.disabled_proof_runtime_executes", + "support": [ + { + "adapter": "model", + "missing_invariants": [], + "result_statuses": [ + "passed" + ], + "scenario_ids": [ + "generated_output_and_proof_runtime_gating_seed_29", + "generated_output_and_proof_runtime_gating_seed_7", + "model_proof_runtime_gating_allowed_outputs" + ], + "status": "passed" + }, + { + "adapter": "real_coordinator", + "missing_invariants": [ + "proof_runtime.no_smt_when_disabled" + ], + "result_statuses": [ + "passed" + ], + "scenario_ids": [ + "real_disabled_proof_runtime_skips_autonomous_checkpoint" + ], + "status": "uncovered" + } + ], + "supporting_invariants": [ + "proof_runtime.no_lean_when_disabled", + "proof_runtime.no_smt_when_disabled" + ], + "title": "Disabled proof runtime executes tools" + }, + { + "description": "Allowed-output routing silently writes papers or strands a proof-only handoff.", + "existing_scenarios": [ + "generated_allowed_outputs_provider_pause_resume_seed_29", + "generated_allowed_outputs_provider_pause_resume_seed_7", + "generated_output_and_proof_runtime_gating_seed_29", + "generated_output_and_proof_runtime_gating_seed_7", + "model_allowed_outputs_provider_pause_stop_resume", + "model_proof_runtime_gating_allowed_outputs", + "real_autonomous_proofs_only_handoff_returns_to_topic_exploration" + ], + "gap_score": 6, + "missing_isolated_artifacts": [ + "real_coordinator:outputs.at_least_one_output_enabled" + ], + "priority": "critical", + "rank": 8, + "required_adapters": [ + "model", + "real_coordinator" + ], + "required_fields": [ + "allowed_outputs", + "proof_runtime_gating", + "workflow_filesystem_state" + ], + "risk_id": "risk.proofs_only_enters_paper", + "support": [ + { + "adapter": "model", + "missing_invariants": [], + "result_statuses": [ + "passed" + ], + "scenario_ids": [ + "generated_allowed_outputs_provider_pause_resume_seed_29", + "generated_allowed_outputs_provider_pause_resume_seed_7", + "generated_output_and_proof_runtime_gating_seed_29", + "generated_output_and_proof_runtime_gating_seed_7", + "model_allowed_outputs_provider_pause_stop_resume", + "model_proof_runtime_gating_allowed_outputs" + ], + "status": "passed" + }, + { + "adapter": "real_coordinator", + "missing_invariants": [ + "outputs.at_least_one_output_enabled" + ], + "result_statuses": [ + "passed" + ], + "scenario_ids": [ + "real_autonomous_proofs_only_handoff_returns_to_topic_exploration" + ], + "status": "uncovered" + } + ], + "supporting_invariants": [ + "outputs.at_least_one_output_enabled", + "outputs.proofs_only_no_paper_phase" + ], + "title": "Proofs-only run enters paper phases" + }, + { + "description": "Clear/reset fails to remove active state or removes preserved run history.", + "existing_scenarios": [ + "generated_manual_scope_clear_archive_seed_29", + "generated_manual_scope_clear_archive_seed_7", + "model_manual_proof_clear_scope_assistant", + "real_manual_archive_and_leanoj_clear_preserve_scope_contract" + ], + "gap_score": 0, + "missing_isolated_artifacts": [], + "priority": "high", + "rank": 9, + "required_adapters": [ + "model", + "real_coordinator" + ], + "required_fields": [ + "workflow_filesystem_state", + "proof_scope_isolation", + "assistant_memory" + ], + "risk_id": "risk.clear_destroys_history", + "support": [ + { + "adapter": "model", + "missing_invariants": [], + "result_statuses": [ + "passed" + ], + "scenario_ids": [ + "generated_manual_scope_clear_archive_seed_29", + "generated_manual_scope_clear_archive_seed_7", + "model_manual_proof_clear_scope_assistant" + ], + "status": "passed" + }, + { + "adapter": "real_coordinator", + "missing_invariants": [], + "result_statuses": [ + "passed" + ], + "scenario_ids": [ + "real_manual_archive_and_leanoj_clear_preserve_scope_contract" + ], + "status": "passed" + } + ], + "supporting_invariants": [ + "state.clear_removes_active_preserves_history", + "proof_scope.manual_clear_archives_active" + ], + "title": "Clear destroys history or leaves active state" + }, + { + "description": "Overflow activity loses configured/effective route attribution, disappears on reload, duplicates terminal stop activity, or turns a scoped proof overflow into a workflow stop.", + "existing_scenarios": [ + "generated_context_overflow_contract_seed_29", + "generated_context_overflow_contract_seed_7", + "real_autonomous_overflow_terminal_stop_once", + "real_manual_aggregator_overflow_identity_persists_across_reload", + "real_proof_context_overflow_is_scoped_nonfatal" + ], + "gap_score": 0, + "missing_isolated_artifacts": [], + "priority": "critical", + "rank": 10, + "required_adapters": [ + "model", + "real_coordinator" + ], + "required_fields": [ + "websocket_api_contracts", + "workflow_filesystem_state", + "provider_pause_resume", + "proof_runtime_gating" + ], + "risk_id": "risk.context_overflow_activity_loses_identity", + "support": [ + { + "adapter": "model", + "missing_invariants": [], + "result_statuses": [ + "passed" + ], + "scenario_ids": [ + "generated_context_overflow_contract_seed_29", + "generated_context_overflow_contract_seed_7" + ], + "status": "passed" + }, + { + "adapter": "real_coordinator", + "missing_invariants": [], + "result_statuses": [ + "passed" + ], + "scenario_ids": [ + "real_autonomous_overflow_terminal_stop_once", + "real_manual_aggregator_overflow_identity_persists_across_reload", + "real_proof_context_overflow_is_scoped_nonfatal" + ], + "status": "passed" + } + ], + "supporting_invariants": [ + "events.context_overflow_route_identity", + "events.context_overflow_persists_across_reload", + "events.context_overflow_terminal_stop_once", + "events.proof_context_overflow_nonfatal" + ], + "title": "Context overflow loses route identity or lifecycle semantics" + }, + { + "description": "A directly injected source reappears as supplemental evidence, or optional retrieval silently replaces mandatory source text when the configured context is too small.", + "existing_scenarios": [ + "generated_rag_prompt_boundaries_seed_29", + "generated_rag_prompt_boundaries_seed_7", + "real_direct_source_rag_exclusion_matrix", + "real_rigor_mandatory_source_overflow_fails_before_model_or_rag" + ], + "gap_score": 0, + "missing_isolated_artifacts": [], + "priority": "high", + "rank": 11, + "required_adapters": [ + "model", + "real_coordinator" + ], + "required_fields": [ + "prompt_context", + "workflow_filesystem_state", + "proof_runtime_gating" + ], + "risk_id": "risk.direct_source_duplicated_by_rag", + "support": [ + { + "adapter": "model", + "missing_invariants": [], + "result_statuses": [ + "passed" + ], + "scenario_ids": [ + "generated_rag_prompt_boundaries_seed_29", + "generated_rag_prompt_boundaries_seed_7" + ], + "status": "passed" + }, + { + "adapter": "real_coordinator", + "missing_invariants": [], + "result_statuses": [ + "passed" + ], + "scenario_ids": [ + "real_direct_source_rag_exclusion_matrix", + "real_rigor_mandatory_source_overflow_fails_before_model_or_rag" + ], + "status": "passed" + } + ], + "supporting_invariants": [ + "prompt.direct_sources_excluded_from_rag", + "prompt.mandatory_source_overflow_fails_visible" + ], + "title": "Direct source is duplicated or displaced by RAG" + }, + { + "description": "Manual, autonomous, archived, or event proof state is exposed in the wrong scope.", + "existing_scenarios": [ + "generated_manual_scope_clear_archive_seed_29", + "generated_manual_scope_clear_archive_seed_7", + "generated_scoped_registered_proof_events_seed_29", + "generated_scoped_registered_proof_events_seed_7", + "model_manual_proof_clear_scope_assistant", + "model_websocket_proof_scope_events", + "real_leanoj_master_proof_stop_resume_preserves_isolated_state", + "real_manual_archive_and_leanoj_clear_preserve_scope_contract", + "real_manual_compiler_proof_only_uses_rigor_settings_and_manual_scope" + ], + "gap_score": 0, + "missing_isolated_artifacts": [], + "priority": "critical", + "rank": 12, + "required_adapters": [ + "model", + "real_coordinator" + ], + "required_fields": [ + "proof_scope_isolation", + "workflow_filesystem_state", + "websocket_api_contracts" + ], + "risk_id": "risk.proof_scope_leak", + "support": [ + { + "adapter": "model", + "missing_invariants": [], + "result_statuses": [ + "passed" + ], + "scenario_ids": [ + "generated_manual_scope_clear_archive_seed_29", + "generated_manual_scope_clear_archive_seed_7", + "generated_scoped_registered_proof_events_seed_29", + "generated_scoped_registered_proof_events_seed_7", + "model_manual_proof_clear_scope_assistant", + "model_websocket_proof_scope_events" + ], + "status": "passed" + }, + { + "adapter": "real_coordinator", + "missing_invariants": [], + "result_statuses": [ + "passed" + ], + "scenario_ids": [ + "real_leanoj_master_proof_stop_resume_preserves_isolated_state", + "real_manual_archive_and_leanoj_clear_preserve_scope_contract", + "real_manual_compiler_proof_only_uses_rigor_settings_and_manual_scope" + ], + "status": "passed" + } + ], + "supporting_invariants": [ + "proof_scope.manual_not_in_autonomous_current", + "proof_scope.autonomous_events_not_manual", + "proof_scope.manual_clear_archives_active" + ], + "title": "Proof records leak across scopes" + } + ], + "schema_version": 1 +} diff --git a/tests/workflow_cross_field/target_risks.py b/tests/workflow_cross_field/target_risks.py new file mode 100644 index 0000000..7e4ff80 --- /dev/null +++ b/tests/workflow_cross_field/target_risks.py @@ -0,0 +1,301 @@ +from __future__ import annotations + +from dataclasses import dataclass +from typing import Literal + +from tests.workflow_harness.coverage_metadata import ( + AdapterType, + InteractionCoverage, + ResultStatus, +) +from tests.workflow_harness.invariant_catalog import FIRST_BUILD_FIELDS, INVARIANTS_BY_ID + + +TARGET_RISK_SCHEMA_VERSION = 1 +RiskPriority = Literal["critical", "high", "medium"] +SupportStatus = Literal["passed", "failed", "skipped", "blocked", "uncovered"] +PRIORITY_WEIGHT: dict[RiskPriority, int] = {"critical": 3, "high": 2, "medium": 1} + + +@dataclass(frozen=True) +class TargetRisk: + risk_id: str + title: str + priority: RiskPriority + required_fields: tuple[str, ...] + supporting_invariants: tuple[str, ...] + required_adapters: tuple[AdapterType, ...] + description: str + + +@dataclass(frozen=True) +class AdapterSupport: + adapter: AdapterType + status: SupportStatus + scenario_ids: tuple[str, ...] + result_statuses: tuple[ResultStatus, ...] + missing_invariants: tuple[str, ...] + + +@dataclass(frozen=True) +class TargetRiskAnalysis: + risk: TargetRisk + rank: int + gap_score: int + support: tuple[AdapterSupport, ...] + existing_scenarios: tuple[str, ...] + missing_isolated_artifacts: tuple[str, ...] + + +TARGET_RISKS: tuple[TargetRisk, ...] = ( + TargetRisk( + "risk.pruned_paper_context_leak", + "Pruned paper remains in model context", + "high", + ("workflow_filesystem_state", "prompt_context"), + ("state.pruned_papers_excluded_from_context",), + ("model", "real_coordinator"), + "A pruned paper remains available to model or RAG context after removal.", + ), + TargetRisk( + "risk.stale_child_phase_takeover", + "Stale child output overrides parent phase", + "critical", + ("runtime_exclusivity", "workflow_filesystem_state", "websocket_api_contracts"), + ("runtime.parent_action_fences_child_outputs",), + ("model", "real_coordinator"), + "Late child output changes state after a parent or user action takes ownership.", + ), + TargetRisk( + "risk.disabled_proof_runtime_executes", + "Disabled proof runtime executes tools", + "critical", + ("proof_runtime_gating", "allowed_outputs"), + ("proof_runtime.no_lean_when_disabled", "proof_runtime.no_smt_when_disabled"), + ("model", "real_coordinator"), + "Lean or SMT runs despite disabled proof-runtime controls.", + ), + TargetRisk( + "risk.proofs_only_enters_paper", + "Proofs-only run enters paper phases", + "critical", + ("allowed_outputs", "proof_runtime_gating", "workflow_filesystem_state"), + ("outputs.at_least_one_output_enabled", "outputs.proofs_only_no_paper_phase"), + ("model", "real_coordinator"), + "Allowed-output routing silently writes papers or strands a proof-only handoff.", + ), + TargetRisk( + "risk.provider_pause_loses_checkpoint", + "Provider pause loses resumable work", + "critical", + ("provider_pause_resume", "workflow_filesystem_state", "proof_runtime_gating"), + ( + "provider.pause_preserves_checkpoint", + "provider.stop_resume_preserves_pause", + "provider.reset_wakes_without_corrupting_checkpoint", + ), + ("model", "real_coordinator"), + "Credit pause, stop, or reset loses or corrupts the durable proof cursor.", + ), + TargetRisk( + "risk.assistant_memory_crosses_boundary", + "Assistant memory crosses role boundary", + "high", + ("assistant_memory", "prompt_context"), + ( + "assistant.no_validator_injection", + "assistant.disable_clears_live_pack", + "prompt.validator_excludes_assistant_memory", + ), + ("model", "real_coordinator"), + "Assistant proof memory reaches validators or remains live after disable.", + ), + TargetRisk( + "risk.proof_scope_leak", + "Proof records leak across scopes", + "critical", + ("proof_scope_isolation", "workflow_filesystem_state", "websocket_api_contracts"), + ( + "proof_scope.manual_not_in_autonomous_current", + "proof_scope.autonomous_events_not_manual", + "proof_scope.manual_clear_archives_active", + ), + ("model", "real_coordinator"), + "Manual, autonomous, archived, or event proof state is exposed in the wrong scope.", + ), + TargetRisk( + "risk.clear_destroys_history", + "Clear destroys history or leaves active state", + "high", + ("workflow_filesystem_state", "proof_scope_isolation", "assistant_memory"), + ("state.clear_removes_active_preserves_history", "proof_scope.manual_clear_archives_active"), + ("model", "real_coordinator"), + "Clear/reset fails to remove active state or removes preserved run history.", + ), + TargetRisk( + "risk.generated_proofs_reenter_prompts", + "Generated proofs reenter source prompts", + "high", + ("prompt_context", "proof_scope_isolation", "workflow_filesystem_state"), + ( + "prompt.generated_appendices_stripped", + "proof_scope.generated_appendices_stripped_from_prompts", + "prompt.proof_source_context_required", + ), + ("model", "real_coordinator"), + "Generated appendices duplicate into source context or replace mandatory source text.", + ), + TargetRisk( + "risk.hosted_desktop_boundary", + "Hosted mode invokes desktop-only behavior", + "critical", + ( + "proof_runtime_gating", + "websocket_api_contracts", + "runtime_exclusivity", + "allowed_outputs", + ), + ( + "proof_runtime.hosted_proof_settings_unavailable", + "api.hosted_desktop_only_routes_unavailable", + ), + ("model", "real_route"), + "Hosted mode reaches proof settings or another desktop-only route or behavior.", + ), + TargetRisk( + "risk.context_overflow_activity_loses_identity", + "Context overflow loses route identity or lifecycle semantics", + "critical", + ( + "websocket_api_contracts", + "workflow_filesystem_state", + "provider_pause_resume", + "proof_runtime_gating", + ), + ( + "events.context_overflow_route_identity", + "events.context_overflow_persists_across_reload", + "events.context_overflow_terminal_stop_once", + "events.proof_context_overflow_nonfatal", + ), + ("model", "real_coordinator"), + ( + "Overflow activity loses configured/effective route attribution, disappears on reload, " + "duplicates terminal stop activity, or turns a scoped proof overflow into a workflow stop." + ), + ), + TargetRisk( + "risk.direct_source_duplicated_by_rag", + "Direct source is duplicated or displaced by RAG", + "high", + ("prompt_context", "workflow_filesystem_state", "proof_runtime_gating"), + ( + "prompt.direct_sources_excluded_from_rag", + "prompt.mandatory_source_overflow_fails_visible", + ), + ("model", "real_coordinator"), + ( + "A directly injected source reappears as supplemental evidence, or optional retrieval " + "silently replaces mandatory source text when the configured context is too small." + ), + ), +) + + +def validate_target_risks(risks: tuple[TargetRisk, ...] = TARGET_RISKS) -> None: + ids = [risk.risk_id for risk in risks] + if len(ids) != len(set(ids)): + raise AssertionError("Target-risk ids must be unique.") + for risk in risks: + if not risk.risk_id.startswith("risk.") or not risk.title or not risk.description: + raise AssertionError(f"Target risk {risk.risk_id!r} has incomplete identity.") + if not risk.required_fields or not risk.supporting_invariants or not risk.required_adapters: + raise AssertionError(f"{risk.risk_id} has incomplete support requirements.") + unknown_fields = set(risk.required_fields) - FIRST_BUILD_FIELDS + unknown_invariants = set(risk.supporting_invariants) - set(INVARIANTS_BY_ID) + if unknown_fields: + raise AssertionError(f"{risk.risk_id} references unknown fields {sorted(unknown_fields)!r}.") + if unknown_invariants: + raise AssertionError( + f"{risk.risk_id} references unknown invariants {sorted(unknown_invariants)!r}." + ) + recognized_fields: set[str] = set() + for invariant_id in risk.supporting_invariants: + invariant = INVARIANTS_BY_ID[invariant_id] + recognized_fields.add(invariant.field) + recognized_fields.update(invariant.crossed_fields) + if not set(risk.required_fields).issubset(recognized_fields): + raise AssertionError(f"{risk.risk_id} has fields disconnected from its invariants.") + + +def analyze_target_risks( + records: tuple[InteractionCoverage, ...] | list[InteractionCoverage], + risks: tuple[TargetRisk, ...] = TARGET_RISKS, +) -> tuple[TargetRiskAnalysis, ...]: + validate_target_risks(risks) + provisional: list[tuple[TargetRisk, int, tuple[AdapterSupport, ...]]] = [] + for risk in risks: + support = tuple(_adapter_support(risk, adapter, records) for adapter in risk.required_adapters) + gap_score = PRIORITY_WEIGHT[risk.priority] * sum( + len(item.missing_invariants) + (1 if item.status != "passed" else 0) + for item in support + ) + provisional.append((risk, gap_score, support)) + + ordered = sorted(provisional, key=lambda item: (-item[1], item[0].risk_id)) + analyses: list[TargetRiskAnalysis] = [] + for rank, (risk, gap_score, support) in enumerate(ordered, start=1): + scenarios = tuple(sorted({item for entry in support for item in entry.scenario_ids})) + missing = tuple( + f"{entry.adapter}:{invariant_id}" + for entry in support + for invariant_id in entry.missing_invariants + ) + analyses.append( + TargetRiskAnalysis(risk, rank, gap_score, support, scenarios, missing) + ) + return tuple(analyses) + + +def _adapter_support( + risk: TargetRisk, + adapter: AdapterType, + records: tuple[InteractionCoverage, ...] | list[InteractionCoverage], +) -> AdapterSupport: + matches = [ + record + for record in records + if record.adapter == adapter + and set(record.invariants).intersection(risk.supporting_invariants) + ] + covered = { + invariant_id + for record in matches + if record.result == "passed" + for invariant_id in record.invariants + if invariant_id in risk.supporting_invariants + } + missing = tuple(sorted(set(risk.supporting_invariants) - covered)) + statuses = tuple(sorted({record.result for record in matches})) + status = _combined_status(statuses, missing) + return AdapterSupport( + adapter=adapter, + status=status, + scenario_ids=tuple(sorted({record.scenario_id for record in matches})), + result_statuses=statuses, + missing_invariants=missing, + ) + + +def _combined_status( + statuses: tuple[ResultStatus, ...], missing_invariants: tuple[str, ...] +) -> SupportStatus: + if "failed" in statuses: + return "failed" + if "blocked" in statuses: + return "blocked" + if "skipped" in statuses: + return "skipped" + if statuses and not missing_invariants: + return "passed" + return "uncovered" diff --git a/tests/workflow_cross_field/targets/generated_allowed_outputs_provider_pause_resume.json b/tests/workflow_cross_field/targets/generated_allowed_outputs_provider_pause_resume.json new file mode 100644 index 0000000..8e07bc6 --- /dev/null +++ b/tests/workflow_cross_field/targets/generated_allowed_outputs_provider_pause_resume.json @@ -0,0 +1,26 @@ +{ + "artifact_type": "generation_target", + "fields": [ + "allowed_outputs", + "provider_pause_resume", + "workflow_filesystem_state" + ], + "invariants": [ + "outputs.proofs_only_no_paper_phase", + "provider.pause_preserves_checkpoint", + "provider.stop_resume_preserves_pause", + "provider.reset_wakes_without_corrupting_checkpoint" + ], + "max_steps": 16, + "must_exercise": [ + "provider_pause", + "stop_resume", + "reset_credit" + ], + "result_ids": [ + "generated_allowed_outputs_provider_pause_resume_seed_7", + "generated_allowed_outputs_provider_pause_resume_seed_29" + ], + "schema_version": 1, + "target_id": "generated_allowed_outputs_provider_pause_resume" +} diff --git a/tests/workflow_cross_field/targets/generated_assistant_prompt_context.json b/tests/workflow_cross_field/targets/generated_assistant_prompt_context.json new file mode 100644 index 0000000..a7e40c4 --- /dev/null +++ b/tests/workflow_cross_field/targets/generated_assistant_prompt_context.json @@ -0,0 +1,24 @@ +{ + "artifact_type": "generation_target", + "fields": [ + "assistant_memory", + "prompt_context" + ], + "invariants": [ + "assistant.disable_clears_live_pack", + "assistant.no_validator_injection", + "prompt.validator_excludes_assistant_memory" + ], + "max_steps": 12, + "must_exercise": [ + "assistant_pack", + "prompt_context", + "assistant_disable" + ], + "result_ids": [ + "generated_assistant_prompt_context_seed_7", + "generated_assistant_prompt_context_seed_29" + ], + "schema_version": 1, + "target_id": "generated_assistant_prompt_context" +} diff --git a/tests/workflow_cross_field/targets/generated_autonomous_paper_checkpoint.json b/tests/workflow_cross_field/targets/generated_autonomous_paper_checkpoint.json new file mode 100644 index 0000000..2726e8e --- /dev/null +++ b/tests/workflow_cross_field/targets/generated_autonomous_paper_checkpoint.json @@ -0,0 +1,22 @@ +{ + "artifact_type": "generation_target", + "fields": [ + "runtime_exclusivity", + "workflow_filesystem_state", + "websocket_api_contracts" + ], + "invariants": [ + "runtime.single_active_workflow" + ], + "max_steps": 7, + "must_exercise": [ + "paper_checkpoint", + "start_blocked" + ], + "result_ids": [ + "generated_autonomous_paper_checkpoint_seed_7", + "generated_autonomous_paper_checkpoint_seed_29" + ], + "schema_version": 1, + "target_id": "generated_autonomous_paper_checkpoint" +} diff --git a/tests/workflow_cross_field/targets/generated_context_overflow_contract.json b/tests/workflow_cross_field/targets/generated_context_overflow_contract.json new file mode 100644 index 0000000..e3e5682 --- /dev/null +++ b/tests/workflow_cross_field/targets/generated_context_overflow_contract.json @@ -0,0 +1,26 @@ +{ + "artifact_type": "generation_target", + "fields": [ + "websocket_api_contracts", + "workflow_filesystem_state", + "provider_pause_resume", + "runtime_exclusivity", + "proof_runtime_gating" + ], + "invariants": [ + "events.context_overflow_route_identity", + "events.context_overflow_persists_across_reload", + "events.context_overflow_terminal_stop_once", + "events.proof_context_overflow_nonfatal" + ], + "max_steps": 6, + "must_exercise": [ + "context_overflow" + ], + "result_ids": [ + "generated_context_overflow_contract_seed_7", + "generated_context_overflow_contract_seed_29" + ], + "schema_version": 1, + "target_id": "generated_context_overflow_contract" +} diff --git a/tests/workflow_cross_field/targets/generated_leanoj_durable_lifecycle.json b/tests/workflow_cross_field/targets/generated_leanoj_durable_lifecycle.json new file mode 100644 index 0000000..0af7f15 --- /dev/null +++ b/tests/workflow_cross_field/targets/generated_leanoj_durable_lifecycle.json @@ -0,0 +1,26 @@ +{ + "artifact_type": "generation_target", + "fields": [ + "runtime_exclusivity", + "workflow_filesystem_state" + ], + "invariants": [ + "runtime.single_active_workflow", + "provider.stop_resume_preserves_pause" + ], + "max_steps": 10, + "must_exercise": [ + "leanoj_draft", + "leanoj_skip", + "leanoj_force", + "stop_resume", + "leanoj_clear", + "start_blocked" + ], + "result_ids": [ + "generated_leanoj_durable_lifecycle_seed_7", + "generated_leanoj_durable_lifecycle_seed_29" + ], + "schema_version": 1, + "target_id": "generated_leanoj_durable_lifecycle" +} diff --git a/tests/workflow_cross_field/targets/generated_manual_aggregator_lifecycle.json b/tests/workflow_cross_field/targets/generated_manual_aggregator_lifecycle.json new file mode 100644 index 0000000..bca11dc --- /dev/null +++ b/tests/workflow_cross_field/targets/generated_manual_aggregator_lifecycle.json @@ -0,0 +1,22 @@ +{ + "artifact_type": "generation_target", + "fields": [ + "runtime_exclusivity", + "workflow_filesystem_state" + ], + "invariants": [ + "runtime.single_active_workflow" + ], + "max_steps": 6, + "must_exercise": [ + "manual_aggregator_accept", + "manual_aggregator_clear", + "start_blocked" + ], + "result_ids": [ + "generated_manual_aggregator_lifecycle_seed_7", + "generated_manual_aggregator_lifecycle_seed_29" + ], + "schema_version": 1, + "target_id": "generated_manual_aggregator_lifecycle" +} diff --git a/tests/workflow_cross_field/targets/generated_manual_scope_clear_archive.json b/tests/workflow_cross_field/targets/generated_manual_scope_clear_archive.json new file mode 100644 index 0000000..6c44a4a --- /dev/null +++ b/tests/workflow_cross_field/targets/generated_manual_scope_clear_archive.json @@ -0,0 +1,25 @@ +{ + "artifact_type": "generation_target", + "fields": [ + "proof_scope_isolation", + "workflow_filesystem_state", + "assistant_memory" + ], + "invariants": [ + "proof_scope.manual_not_in_autonomous_current", + "proof_scope.manual_clear_archives_active", + "state.clear_removes_active_preserves_history" + ], + "max_steps": 14, + "must_exercise": [ + "manual_proof", + "assistant_pack", + "manual_archive" + ], + "result_ids": [ + "generated_manual_scope_clear_archive_seed_7", + "generated_manual_scope_clear_archive_seed_29" + ], + "schema_version": 1, + "target_id": "generated_manual_scope_clear_archive" +} diff --git a/tests/workflow_cross_field/targets/generated_output_and_proof_runtime_gating.json b/tests/workflow_cross_field/targets/generated_output_and_proof_runtime_gating.json new file mode 100644 index 0000000..1305db2 --- /dev/null +++ b/tests/workflow_cross_field/targets/generated_output_and_proof_runtime_gating.json @@ -0,0 +1,24 @@ +{ + "artifact_type": "generation_target", + "fields": [ + "proof_runtime_gating", + "allowed_outputs" + ], + "invariants": [ + "outputs.at_least_one_output_enabled", + "outputs.papers_only_skips_proof_work", + "proof_runtime.no_lean_when_disabled", + "proof_runtime.no_smt_when_disabled" + ], + "max_steps": 10, + "must_exercise": [ + "no_outputs_rejected", + "papers_only" + ], + "result_ids": [ + "generated_output_and_proof_runtime_gating_seed_7", + "generated_output_and_proof_runtime_gating_seed_29" + ], + "schema_version": 1, + "target_id": "generated_output_and_proof_runtime_gating" +} diff --git a/tests/workflow_cross_field/targets/generated_rag_prompt_boundaries.json b/tests/workflow_cross_field/targets/generated_rag_prompt_boundaries.json new file mode 100644 index 0000000..c7ae41c --- /dev/null +++ b/tests/workflow_cross_field/targets/generated_rag_prompt_boundaries.json @@ -0,0 +1,28 @@ +{ + "artifact_type": "generation_target", + "fields": [ + "prompt_context", + "workflow_filesystem_state", + "proof_runtime_gating", + "websocket_api_contracts" + ], + "invariants": [ + "prompt.user_prompt_direct_injected", + "prompt.proof_source_context_required", + "prompt.direct_sources_excluded_from_rag", + "prompt.mandatory_source_overflow_fails_visible", + "prompt.generated_appendices_stripped" + ], + "max_steps": 8, + "must_exercise": [ + "prompt_context", + "rag_source_exclusion", + "mandatory_source_overflow" + ], + "result_ids": [ + "generated_rag_prompt_boundaries_seed_7", + "generated_rag_prompt_boundaries_seed_29" + ], + "schema_version": 1, + "target_id": "generated_rag_prompt_boundaries" +} diff --git a/tests/workflow_cross_field/targets/generated_runtime_start_exclusivity.json b/tests/workflow_cross_field/targets/generated_runtime_start_exclusivity.json new file mode 100644 index 0000000..267e951 --- /dev/null +++ b/tests/workflow_cross_field/targets/generated_runtime_start_exclusivity.json @@ -0,0 +1,20 @@ +{ + "artifact_type": "generation_target", + "fields": [ + "runtime_exclusivity", + "websocket_api_contracts" + ], + "invariants": [ + "runtime.single_active_workflow" + ], + "max_steps": 8, + "must_exercise": [ + "start_blocked" + ], + "result_ids": [ + "generated_runtime_start_exclusivity_seed_7", + "generated_runtime_start_exclusivity_seed_29" + ], + "schema_version": 1, + "target_id": "generated_runtime_start_exclusivity" +} diff --git a/tests/workflow_cross_field/targets/generated_scoped_registered_proof_events.json b/tests/workflow_cross_field/targets/generated_scoped_registered_proof_events.json new file mode 100644 index 0000000..c7c1f61 --- /dev/null +++ b/tests/workflow_cross_field/targets/generated_scoped_registered_proof_events.json @@ -0,0 +1,23 @@ +{ + "artifact_type": "generation_target", + "fields": [ + "websocket_api_contracts", + "proof_scope_isolation" + ], + "invariants": [ + "events.frontend_events_include_scope_phase", + "events.proof_verified_after_registration", + "proof_scope.autonomous_events_not_manual" + ], + "max_steps": 10, + "must_exercise": [ + "scoped_event", + "registered_proof_event" + ], + "result_ids": [ + "generated_scoped_registered_proof_events_seed_7", + "generated_scoped_registered_proof_events_seed_29" + ], + "schema_version": 1, + "target_id": "generated_scoped_registered_proof_events" +} diff --git a/tests/workflow_cross_field/test_artifacts.py b/tests/workflow_cross_field/test_artifacts.py new file mode 100644 index 0000000..ada3fea --- /dev/null +++ b/tests/workflow_cross_field/test_artifacts.py @@ -0,0 +1,158 @@ +from __future__ import annotations + +import json + +import pytest + +from tests.workflow_cross_field.artifacts import ( + RESULTS_DIR, + SCENARIOS_DIR, + SCHEMA_VERSION, + SEEDS, + SUMMARY_PATH, + TARGETS_DIR, + TARGET_RISKS_JSON_PATH, + TARGET_RISKS_MARKDOWN_PATH, + render_artifacts, +) +from tests.workflow_cross_field.support_graph import ( + SUPPORT_GRAPH_JSON_PATH, + SUPPORT_GRAPH_MARKDOWN_PATH, +) +from tests.workflow_harness.coverage_metadata import ( + InteractionCoverage, + assert_coverage_metadata_valid, +) + + +def test_projection_is_deterministic_and_complete(): + rendered = render_artifacts() + checked_in = { + path: path.read_bytes() + for path in ( + list(SCENARIOS_DIR.glob("*.json")) + + list(TARGETS_DIR.glob("*.json")) + + list(RESULTS_DIR.glob("*.json")) + + [ + SUMMARY_PATH, + TARGET_RISKS_JSON_PATH, + TARGET_RISKS_MARKDOWN_PATH, + SUPPORT_GRAPH_JSON_PATH, + SUPPORT_GRAPH_MARKDOWN_PATH, + ] + ) + } + assert rendered == checked_in + assert list(SCENARIOS_DIR.glob("*.json")) + assert "| browser_smoke |" in SUMMARY_PATH.read_text(encoding="utf-8") + assert "uncovered" in SUMMARY_PATH.read_text(encoding="utf-8") + + +def test_generated_results_have_replay_diagnostics_and_evidence(): + for path in RESULTS_DIR.glob("*.json"): + record = json.loads(path.read_text(encoding="utf-8")) + assert record["schema_version"] == SCHEMA_VERSION + assert record["artifact_type"] == "result" + if record["result"] == "passed": + assert record["evidence"] + else: + assert record["diagnostics"]["reason"] + if "target_id" in record: + assert record["seed"] in SEEDS + assert record["replay"] + assert record["diagnostics"]["final_mode"] + assert record["diagnostics"]["final_phase"] + assert set(record["invariants"]) == set(record["exercised_invariants"]) + + +def test_generated_targets_are_projected_and_linked_bidirectionally(): + target_paths = list(TARGETS_DIR.glob("*.json")) + assert target_paths + for path in target_paths: + target = json.loads(path.read_text(encoding="utf-8")) + assert target["schema_version"] == SCHEMA_VERSION + assert target["artifact_type"] == "generation_target" + assert target["target_id"] == path.stem + assert target["fields"] + assert target["invariants"] + assert target["must_exercise"] + assert target["result_ids"] == [ + f"{target['target_id']}_seed_{seed}" for seed in SEEDS + ] + for result_id in target["result_ids"]: + result = json.loads( + (RESULTS_DIR / f"{result_id}.json").read_text(encoding="utf-8") + ) + assert result["target_id"] == target["target_id"] + assert result["target_artifact"] == f"targets/{path.name}" + assert result["scenario_id"] == result_id + + +def test_disk_artifacts_link_to_known_scenarios_and_catalog(): + from tests.workflow_browser_smoke.coverage_records import BROWSER_SMOKE_COVERAGE + from tests.workflow_harness.cross_field_scenarios import CROSS_FIELD_SCENARIOS + from tests.workflow_harness.invariant_catalog import INVARIANTS_BY_ID + from tests.workflow_real_adapters.coverage_records import REAL_ADAPTER_COVERAGE + + known = {item.scenario_id for item in CROSS_FIELD_SCENARIOS} + known.update(item.scenario_id for item in REAL_ADAPTER_COVERAGE) + known.update(item.scenario_id for item in BROWSER_SMOKE_COVERAGE) + for path in SCENARIOS_DIR.glob("*.json"): + scenario = json.loads(path.read_text(encoding="utf-8")) + assert scenario["schema_version"] == SCHEMA_VERSION + assert scenario["artifact_type"] == "scenario" + assert scenario["scenario_id"] in known + assert set(scenario["invariants"]).issubset(INVARIANTS_BY_ID) + assert (RESULTS_DIR / path.name).exists() + for path in RESULTS_DIR.glob("*.json"): + result = json.loads(path.read_text(encoding="utf-8")) + assert set(result["invariants"]).issubset(INVARIANTS_BY_ID) + if "target_id" not in result: + assert result["scenario_id"] in known + + +def test_target_risk_artifacts_are_versioned_ranked_and_linked(): + payload = json.loads(TARGET_RISKS_JSON_PATH.read_text(encoding="utf-8")) + assert payload["schema_version"] == 1 + assert payload["artifact_type"] == "target_risk_analysis" + assert len(payload["risks"]) == 12 + assert [risk["rank"] for risk in payload["risks"]] == list(range(1, 13)) + assert [(-risk["gap_score"], risk["risk_id"]) for risk in payload["risks"]] == sorted( + (-risk["gap_score"], risk["risk_id"]) for risk in payload["risks"] + ) + for risk in payload["risks"]: + assert risk["required_fields"] + assert risk["supporting_invariants"] + assert risk["required_adapters"] + assert {item["adapter"] for item in risk["support"]} == set(risk["required_adapters"]) + assert all( + item["status"] in {"passed", "failed", "skipped", "blocked", "uncovered"} + for item in risk["support"] + ) + markdown = TARGET_RISKS_MARKDOWN_PATH.read_text(encoding="utf-8") + assert "Missing isolated artifacts" in markdown + assert "real_coordinator: uncovered" in markdown + + +@pytest.mark.parametrize( + ("changes", "message"), + [ + ({"seed": -1, "replay": ("start",), "evidence": ("proof",)}, "negative seed"), + ({"seed": 1, "evidence": ("proof",)}, "seed but no replay"), + ({"evidence": ()}, "passed without evidence"), + ({"result": "blocked", "evidence": ()}, "without diagnostics"), + ], +) +def test_extended_coverage_metadata_rejects_incomplete_records(changes, message): + values = { + "scenario_id": "validation_fixture", + "fields": ("allowed_outputs",), + "invariants": ("outputs.at_least_one_output_enabled",), + "adapter": "model", + "result": "passed", + "test_file": "tests/workflow_cross_field/test_artifacts.py", + "evidence": ("proof",), + } + values.update(changes) + with pytest.raises(AssertionError, match=message): + assert_coverage_metadata_valid(InteractionCoverage(**values)) diff --git a/tests/workflow_cross_field/test_support_graph.py b/tests/workflow_cross_field/test_support_graph.py new file mode 100644 index 0000000..3b985dc --- /dev/null +++ b/tests/workflow_cross_field/test_support_graph.py @@ -0,0 +1,94 @@ +from __future__ import annotations + +import json + +import pytest + +from tests.workflow_cross_field.support_graph import ( + SUPPORT_GRAPH_SCHEMA_VERSION, + build_support_graph, + render_support_graph_json, + render_support_graph_markdown, + validate_support_graph, +) +from tests.workflow_harness.coverage_metadata import InteractionCoverage +from tests.workflow_real_adapters.coverage_records import REAL_ADAPTER_COVERAGE + + +def test_support_graph_is_deterministic_across_record_order(): + forward_json = render_support_graph_json(REAL_ADAPTER_COVERAGE) + reverse_json = render_support_graph_json(reversed(REAL_ADAPTER_COVERAGE)) + forward_markdown = render_support_graph_markdown(REAL_ADAPTER_COVERAGE) + reverse_markdown = render_support_graph_markdown(reversed(REAL_ADAPTER_COVERAGE)) + + assert forward_json == reverse_json + assert forward_markdown == reverse_markdown + assert json.loads(forward_json)["schema_version"] == SUPPORT_GRAPH_SCHEMA_VERSION + + +def test_support_graph_separates_declared_and_observed_basis(): + graph = build_support_graph(REAL_ADAPTER_COVERAGE) + + hosted = [ + edge + for edge in graph["edges"] + if edge["invariant_id"] == "proof_runtime.hosted_proof_settings_unavailable" + and edge["adapter"] == "real_route" + ] + parent_fence = [ + edge + for edge in graph["edges"] + if edge["invariant_id"] == "runtime.parent_action_fences_child_outputs" + and edge["adapter"] == "real_coordinator" + ] + + assert any(edge["observed"] and edge["status"] == "passed" for edge in hosted) + assert any( + edge["observed"] + and not edge["declared"] + and edge["status"] == "blocked" + and edge["reason"] + for edge in parent_fence + ) + + +def test_support_graph_emits_structured_required_adapter_gaps(): + graph = build_support_graph(()) + + assert graph["gaps"] + assert all(gap["status"] == "uncovered" for gap in graph["gaps"]) + assert all(gap["gap_id"] and gap["reason"] for gap in graph["gaps"]) + + +def test_support_graph_preserves_failed_skipped_and_blocked_statuses(): + records = tuple( + InteractionCoverage( + scenario_id=f"status-{status}", + fields=("proof_runtime_gating",), + invariants=("proof_runtime.no_lean_when_disabled",), + adapter="model", + result=status, + test_file="tests/workflow_cross_field/test_support_graph.py", + diagnostics={"reason": f"{status} reason"} if status != "passed" else None, + evidence=("observed",) if status == "passed" else (), + ) + for status in ("passed", "failed", "skipped", "blocked") + ) + + graph = build_support_graph(records) + statuses = { + edge["status"] + for edge in graph["edges"] + if edge["scenario_id"] and edge["scenario_id"].startswith("status-") + } + + assert statuses == {"passed", "failed", "skipped", "blocked"} + + +def test_support_graph_validation_rejects_false_observation(): + graph = build_support_graph(REAL_ADAPTER_COVERAGE) + graph["edges"][0]["observed"] = True + graph["edges"][0]["scenario_id"] = None + + with pytest.raises(AssertionError, match="Observed support edge"): + validate_support_graph(graph) diff --git a/tests/workflow_cross_field/test_target_risks.py b/tests/workflow_cross_field/test_target_risks.py new file mode 100644 index 0000000..2ddb648 --- /dev/null +++ b/tests/workflow_cross_field/test_target_risks.py @@ -0,0 +1,101 @@ +from __future__ import annotations + +from dataclasses import replace + +import pytest + +from tests.workflow_cross_field.target_risks import ( + TARGET_RISKS, + analyze_target_risks, + validate_target_risks, +) +from tests.workflow_harness.coverage_metadata import InteractionCoverage + + +def _record( + scenario_id: str, + invariant: str, + *, + adapter: str = "model", + result: str = "passed", +) -> InteractionCoverage: + return InteractionCoverage( + scenario_id=scenario_id, + fields=("runtime_exclusivity", "workflow_filesystem_state"), + invariants=(invariant,), + adapter=adapter, + result=result, + test_file="tests/workflow_cross_field/test_target_risks.py", + diagnostics={"reason": "fixture"} if result != "passed" else None, + evidence=("fixture",) if result == "passed" else (), + ) + + +def test_catalog_has_exactly_twelve_unique_recognized_risks(): + validate_target_risks() + assert len(TARGET_RISKS) == 12 + risk_ids = {risk.risk_id for risk in TARGET_RISKS} + assert len(risk_ids) == 12 + assert "risk.pruned_paper_context_leak" in risk_ids + assert "risk.hosted_desktop_boundary" in risk_ids + assert "risk.context_overflow_activity_loses_identity" in risk_ids + assert "risk.direct_source_duplicated_by_rag" in risk_ids + assert "risk.concurrent_workflow_ownership" not in risk_ids + assert "risk.proof_event_precedes_registration" not in risk_ids + + +@pytest.mark.parametrize( + "change, message", + [ + ({"risk_id": ""}, "incomplete identity"), + ({"required_fields": ("unknown",)}, "unknown fields"), + ({"supporting_invariants": ("unknown",)}, "unknown invariants"), + ({"required_fields": ("assistant_memory",)}, "disconnected"), + ], +) +def test_catalog_validation_rejects_unrecognized_or_disconnected_values(change, message): + risk = replace(TARGET_RISKS[0], **change) + with pytest.raises(AssertionError, match=message): + validate_target_risks((risk,)) + + +def test_support_preserves_adapter_and_result_status_distinctions(): + risk = next(item for item in TARGET_RISKS if item.risk_id == "risk.provider_pause_loses_checkpoint") + records = ( + _record("model_pass", risk.supporting_invariants[0]), + _record( + "coordinator_block", + risk.supporting_invariants[0], + adapter="real_coordinator", + result="blocked", + ), + ) + analysis = analyze_target_risks(records, (risk,))[0] + by_adapter = {item.adapter: item for item in analysis.support} + assert by_adapter["model"].status == "uncovered" + assert by_adapter["model"].result_statuses == ("passed",) + assert by_adapter["real_coordinator"].status == "blocked" + assert by_adapter["real_coordinator"].result_statuses == ("blocked",) + assert analysis.missing_isolated_artifacts + + +def test_complete_passed_support_removes_adapter_gap(): + risk = TARGET_RISKS[0] + records = tuple( + _record(f"{adapter}_{index}", invariant, adapter=adapter) + for adapter in risk.required_adapters + for index, invariant in enumerate(risk.supporting_invariants) + ) + analysis = analyze_target_risks(records, (risk,))[0] + assert all(item.status == "passed" for item in analysis.support) + assert analysis.missing_isolated_artifacts == () + assert analysis.gap_score == 0 + + +def test_ranking_is_deterministic_with_risk_id_tiebreak(): + first = analyze_target_risks((), TARGET_RISKS) + second = analyze_target_risks((), tuple(reversed(TARGET_RISKS))) + assert [item.risk.risk_id for item in first] == [item.risk.risk_id for item in second] + assert [(item.rank, item.gap_score) for item in first] == [ + (item.rank, item.gap_score) for item in second + ] diff --git a/tests/workflow_harness/__init__.py b/tests/workflow_harness/__init__.py new file mode 100644 index 0000000..8fa1dd7 --- /dev/null +++ b/tests/workflow_harness/__init__.py @@ -0,0 +1,2 @@ +"""Reusable workflow-state harness for recombinatory tests.""" + diff --git a/tests/workflow_harness/actions.py b/tests/workflow_harness/actions.py new file mode 100644 index 0000000..32cc5ed --- /dev/null +++ b/tests/workflow_harness/actions.py @@ -0,0 +1,210 @@ +from __future__ import annotations + +from collections.abc import Callable + +from .invariants import assert_all_invariants +from .model import WorkflowModel + + +WorkflowAction = Callable[[WorkflowModel], None] + + +def run_actions(model: WorkflowModel, actions: list[WorkflowAction]) -> WorkflowModel: + for action in actions: + action(model) + assert_all_invariants(model) + return model + + +def start_autonomous_proofs_only(model: WorkflowModel) -> None: + model.start_autonomous(proofs=True, papers=False, lean_enabled=True) + + +def start_autonomous_papers_and_proofs(model: WorkflowModel) -> None: + model.start_autonomous(proofs=True, papers=True, lean_enabled=True) + + +def start_autonomous_papers_only(model: WorkflowModel) -> None: + model.start_autonomous(proofs=False, papers=True, lean_enabled=False) + + +def start_autonomous_no_outputs(model: WorkflowModel) -> None: + model.start_autonomous_with_no_outputs() + + +def start_manual_compiler(model: WorkflowModel) -> None: + model.start_manual_compiler() + + +def start_manual_aggregator(model: WorkflowModel) -> None: + model.start_manual_aggregator() + + +def accept_manual_aggregator_submission(model: WorkflowModel) -> None: + model.accept_manual_aggregator_submission() + + +def start_leanoj(model: WorkflowModel) -> None: + model.start_leanoj() + + +def edit_leanoj_master_proof(model: WorkflowModel) -> None: + model.edit_leanoj_master_proof() + + +def skip_leanoj_brainstorm(model: WorkflowModel) -> None: + model.skip_leanoj_brainstorm() + + +def force_leanoj_brainstorm(model: WorkflowModel) -> None: + model.force_leanoj_brainstorm() + + +def enter_autonomous_paper_checkpoint(model: WorkflowModel) -> None: + model.enter_autonomous_paper_checkpoint() + + +def complete_autonomous_paper_checkpoint(model: WorkflowModel) -> None: + model.complete_autonomous_paper_checkpoint() + + +def attempt_conflict_during_autonomous_paper_checkpoint(model: WorkflowModel) -> None: + model.start_manual_compiler() + + +def enable_manual_lean(model: WorkflowModel) -> None: + model.record("enable_manual_lean") + model.lean.enabled = True + + +def complete_topic_exploration(model: WorkflowModel) -> None: + model.complete_topic_exploration() + + +def complete_brainstorm(model: WorkflowModel) -> None: + model.complete_brainstorm() + + +def attempt_disabled_lean_checkpoint(model: WorkflowModel) -> None: + model.attempt_disabled_lean_checkpoint() + + +def start_child_task(model: WorkflowModel) -> None: + model.start_child_task() + + +def stale_child_output_arrives_after_parent_action(model: WorkflowModel) -> None: + model.stale_child_output_arrives_after_parent_action() + + +def force_paper_writing(model: WorkflowModel) -> None: + model.force_paper_writing() + + +def complete_paper(model: WorkflowModel) -> None: + model.complete_paper() + + +def run_manual_proof_check(model: WorkflowModel) -> None: + model.run_manual_proof_check() + + +def simulate_credit_exhaustion(model: WorkflowModel) -> None: + model.simulate_credit_exhaustion() + + +def run_smt_hint_generation(model: WorkflowModel) -> None: + model.run_smt_hint_generation() + + +def simulate_provider_output_truncation(model: WorkflowModel) -> None: + model.simulate_provider_output_truncation() + + +def try_hosted_desktop_only_route(model: WorkflowModel) -> None: + model.try_hosted_desktop_only_route() + + +def reset_credit(model: WorkflowModel) -> None: + model.reset_credit() + + +def stop(model: WorkflowModel) -> None: + model.stop() + + +def resume(model: WorkflowModel) -> None: + model.resume() + + +def clear(model: WorkflowModel) -> None: + model.clear() + + +def disable_session_history(model: WorkflowModel) -> None: + model.toggle_session_history_memory(False) + + +def enable_session_history(model: WorkflowModel) -> None: + model.toggle_session_history_memory(True) + + +def refresh_assistant_pack(model: WorkflowModel) -> None: + model.refresh_assistant_pack("prior-proof-1", "prior-proof-2") + + +def assistant_retrieve_non_blocking(model: WorkflowModel) -> None: + model.assistant_retrieve_non_blocking() + + +def assistant_stagnant_pack_backoff(model: WorkflowModel) -> None: + model.assistant_stagnant_pack_backoff() + + +def complete_brainstorm_with_generated_paper(model: WorkflowModel) -> None: + model.complete_brainstorm_with_generated_paper() + + +def resume_completed_brainstorm(model: WorkflowModel) -> None: + model.resume_completed_brainstorm() + + +def prune_paper(model: WorkflowModel) -> None: + model.prune_paper("paper-1") + + +def add_pruned_paper_to_model_context(model: WorkflowModel) -> None: + model.add_paper_to_model_context("paper-1") + + +def prepare_prompt_context(model: WorkflowModel) -> None: + model.prepare_prompt_context() + + +def prepare_validator_prompt_with_live_assistant(model: WorkflowModel) -> None: + model.prepare_validator_prompt_with_live_assistant() + + +def resolve_runtime_path(model: WorkflowModel) -> None: + model.resolve_runtime_path() + + +def verify_rag_source_exclusion(model: WorkflowModel) -> None: + model.verify_rag_source_exclusion() + + +def reject_mandatory_source_overflow(model: WorkflowModel) -> None: + model.reject_mandatory_source_overflow() + + +def emit_frontend_scoped_event(model: WorkflowModel) -> None: + model.emit_frontend_scoped_event() + + +def emit_context_overflow_contract_events(model: WorkflowModel) -> None: + model.emit_context_overflow_contract_events() + + +def emit_registered_proof_verified(model: WorkflowModel) -> None: + model.emit_registered_proof_verified() + diff --git a/tests/workflow_harness/coverage_metadata.py b/tests/workflow_harness/coverage_metadata.py new file mode 100644 index 0000000..91fff4f --- /dev/null +++ b/tests/workflow_harness/coverage_metadata.py @@ -0,0 +1,106 @@ +from __future__ import annotations + +from dataclasses import dataclass +from typing import Literal, Mapping + +from .invariant_catalog import INVARIANTS_BY_ID + + +AdapterType = Literal["model", "real_route", "real_coordinator", "browser_smoke"] +ResultStatus = Literal["passed", "failed", "skipped", "blocked"] +CoverageRunner = Literal["pytest", "playwright"] + + +@dataclass(frozen=True) +class InteractionCoverage: + scenario_id: str + fields: tuple[str, ...] + invariants: tuple[str, ...] + adapter: AdapterType + result: ResultStatus + test_file: str + seed: int | None = None + replay: tuple[str, ...] = () + diagnostics: Mapping[str, str] | None = None + evidence: tuple[str, ...] = () + runner: CoverageRunner | None = None + test_selectors: tuple[str, ...] = () + asserted_invariants: tuple[str, ...] = () + + +def assert_coverage_metadata_valid(record: InteractionCoverage) -> None: + if not record.scenario_id: + raise AssertionError("Coverage metadata is missing scenario_id.") + if not record.fields: + raise AssertionError(f"{record.scenario_id} does not declare crossed fields.") + if not record.invariants: + raise AssertionError(f"{record.scenario_id} does not declare invariant ids.") + if not record.test_file.startswith("tests/"): + raise AssertionError(f"{record.scenario_id} has non-test file path {record.test_file!r}.") + if record.seed is not None and record.seed < 0: + raise AssertionError(f"{record.scenario_id} has a negative seed.") + if record.seed is not None and not record.replay: + raise AssertionError(f"{record.scenario_id} has a seed but no replay.") + if record.result == "passed" and not record.evidence: + raise AssertionError(f"{record.scenario_id} passed without evidence.") + executable_claim = record.adapter in {"real_route", "real_coordinator", "browser_smoke"} + if record.result == "passed" and executable_claim and not record.runner: + raise AssertionError(f"{record.scenario_id} passed without an executable runner.") + if record.result == "passed" and executable_claim and not record.test_selectors: + raise AssertionError(f"{record.scenario_id} passed without exact test selectors.") + if ( + record.result == "passed" + and executable_claim + and set(record.asserted_invariants) != set(record.invariants) + ): + raise AssertionError( + f"{record.scenario_id} must link every claimed invariant to an asserted invariant." + ) + if record.result == "blocked" and ( + record.runner or record.test_selectors or record.asserted_invariants or record.evidence + ): + raise AssertionError(f"{record.scenario_id} is blocked but is not metadata-only.") + if record.result in {"failed", "skipped", "blocked"} and not record.diagnostics: + raise AssertionError( + f"{record.scenario_id} is {record.result} without diagnostics." + ) + if any(not item.strip() for item in record.replay): + raise AssertionError(f"{record.scenario_id} has an empty replay step.") + if any(not item.strip() for item in record.evidence): + raise AssertionError(f"{record.scenario_id} has empty evidence.") + if any(not item.strip() for item in record.test_selectors): + raise AssertionError(f"{record.scenario_id} has an empty test selector.") + if len(record.test_selectors) != len(set(record.test_selectors)): + raise AssertionError(f"{record.scenario_id} has duplicate test selectors.") + if len(record.asserted_invariants) != len(set(record.asserted_invariants)): + raise AssertionError(f"{record.scenario_id} has duplicate asserted invariants.") + if record.runner == "pytest": + for selector in record.test_selectors: + if selector.count("::") != 1 or not selector.split("::", 1)[1].startswith("test_"): + raise AssertionError( + f"{record.scenario_id} has non-exact pytest selector {selector!r}." + ) + if record.runner == "playwright": + for selector in record.test_selectors: + if "::" not in selector or not selector.split("::", 1)[0].endswith(".spec.js"): + raise AssertionError( + f"{record.scenario_id} has non-exact Playwright selector {selector!r}." + ) + + for invariant_id in record.invariants: + if invariant_id not in INVARIANTS_BY_ID: + raise AssertionError(f"{record.scenario_id} references unknown invariant {invariant_id!r}.") + invariant = INVARIANTS_BY_ID[invariant_id] + if record.result != "blocked" and record.adapter not in invariant.adapters: + raise AssertionError( + f"{record.scenario_id} uses adapter {record.adapter!r} for " + f"invariant {invariant_id!r}, but the catalog does not list that adapter." + ) + + +def assert_coverage_records_valid(records: tuple[InteractionCoverage, ...]) -> None: + scenario_ids = [record.scenario_id for record in records] + if len(scenario_ids) != len(set(scenario_ids)): + raise AssertionError("Coverage metadata scenario ids must be unique.") + for record in records: + assert_coverage_metadata_valid(record) diff --git a/tests/workflow_harness/cross_field_scenarios.py b/tests/workflow_harness/cross_field_scenarios.py new file mode 100644 index 0000000..a5b0a65 --- /dev/null +++ b/tests/workflow_harness/cross_field_scenarios.py @@ -0,0 +1,172 @@ +from __future__ import annotations + +from collections.abc import Callable +from dataclasses import dataclass + +from . import actions +from .coverage_metadata import AdapterType, InteractionCoverage, ResultStatus +from .invariant_catalog import FIRST_BUILD_FIELDS, INVARIANTS_BY_ID +from .model import WorkflowModel + + +ScenarioAction = Callable[[WorkflowModel], None] + + +@dataclass(frozen=True) +class CrossFieldScenario: + scenario_id: str + fields: tuple[str, ...] + invariants: tuple[str, ...] + adapter: AdapterType + actions: tuple[ScenarioAction, ...] + must_exercise: tuple[str, ...] = () + expected_result: ResultStatus = "passed" + notes: str = "" + + def coverage_record(self, test_file: str) -> InteractionCoverage: + return InteractionCoverage( + scenario_id=self.scenario_id, + fields=self.fields, + invariants=self.invariants, + adapter=self.adapter, + result=self.expected_result, + test_file=test_file, + replay=tuple(action.__name__ for action in self.actions), + evidence=self.must_exercise or self.invariants, + ) + + +def validate_cross_field_scenario(scenario: CrossFieldScenario) -> None: + if not scenario.scenario_id: + raise AssertionError("Cross-field scenario is missing scenario_id.") + if len(scenario.fields) < 2: + raise AssertionError(f"{scenario.scenario_id} must cross at least two fields.") + if len(set(scenario.fields)) != len(scenario.fields): + raise AssertionError(f"{scenario.scenario_id} declares duplicate fields.") + if not scenario.invariants: + raise AssertionError(f"{scenario.scenario_id} does not declare invariants.") + if scenario.expected_result == "passed" and not scenario.actions: + raise AssertionError(f"{scenario.scenario_id} has no executable actions.") + + for field in scenario.fields: + if field not in FIRST_BUILD_FIELDS: + raise AssertionError(f"{scenario.scenario_id} references unknown field {field!r}.") + catalog_fields_for_invariants: set[str] = set() + for invariant_id in scenario.invariants: + if invariant_id not in INVARIANTS_BY_ID: + raise AssertionError( + f"{scenario.scenario_id} references unknown invariant {invariant_id!r}." + ) + invariant = INVARIANTS_BY_ID[invariant_id] + catalog_fields_for_invariants.add(invariant.field) + catalog_fields_for_invariants.update(invariant.crossed_fields) + if scenario.adapter not in invariant.adapters and scenario.expected_result != "blocked": + raise AssertionError( + f"{scenario.scenario_id} uses adapter {scenario.adapter!r} for " + f"invariant {invariant_id!r}, but the catalog does not list that adapter." + ) + unmatched_fields = set(scenario.fields) - catalog_fields_for_invariants + if unmatched_fields: + raise AssertionError( + f"{scenario.scenario_id} declares fields not connected to its invariants: " + f"{sorted(unmatched_fields)!r}." + ) + + +CROSS_FIELD_SCENARIOS: tuple[CrossFieldScenario, ...] = ( + CrossFieldScenario( + scenario_id="model_allowed_outputs_provider_pause_stop_resume", + fields=("allowed_outputs", "provider_pause_resume", "workflow_filesystem_state"), + invariants=( + "outputs.proofs_only_no_paper_phase", + "provider.pause_preserves_checkpoint", + "provider.stop_resume_preserves_pause", + "provider.reset_wakes_without_corrupting_checkpoint", + ), + adapter="model", + actions=( + actions.start_autonomous_proofs_only, + actions.complete_topic_exploration, + actions.simulate_credit_exhaustion, + actions.complete_brainstorm, + actions.stop, + actions.resume, + actions.reset_credit, + ), + must_exercise=("provider_pause", "stop_resume", "reset_credit"), + ), + CrossFieldScenario( + scenario_id="model_assistant_prompt_validator_exclusion", + fields=("assistant_memory", "prompt_context"), + invariants=( + "assistant.no_validator_injection", + "assistant.disable_clears_live_pack", + "prompt.validator_excludes_assistant_memory", + ), + adapter="model", + actions=( + actions.start_autonomous_papers_and_proofs, + actions.refresh_assistant_pack, + actions.prepare_prompt_context, + actions.disable_session_history, + ), + must_exercise=("assistant_pack", "assistant_disable", "prompt_context"), + ), + CrossFieldScenario( + scenario_id="model_manual_proof_clear_scope_assistant", + fields=("proof_scope_isolation", "workflow_filesystem_state", "assistant_memory"), + invariants=( + "proof_scope.manual_not_in_autonomous_current", + "proof_scope.manual_clear_archives_active", + "state.clear_removes_active_preserves_history", + "assistant.disable_clears_live_pack", + ), + adapter="model", + actions=( + actions.start_manual_compiler, + actions.enable_manual_lean, + actions.run_manual_proof_check, + actions.refresh_assistant_pack, + actions.clear, + ), + must_exercise=("manual_proof", "manual_archive", "assistant_pack"), + ), + CrossFieldScenario( + scenario_id="model_websocket_proof_scope_events", + fields=("websocket_api_contracts", "proof_scope_isolation"), + invariants=( + "events.frontend_events_include_scope_phase", + "events.proof_verified_after_registration", + "proof_scope.autonomous_events_not_manual", + ), + adapter="model", + actions=( + actions.start_autonomous_papers_and_proofs, + actions.emit_frontend_scoped_event, + actions.emit_registered_proof_verified, + ), + must_exercise=("scoped_event", "registered_proof_event"), + ), + CrossFieldScenario( + scenario_id="model_proof_runtime_gating_allowed_outputs", + fields=("proof_runtime_gating", "allowed_outputs"), + invariants=( + "outputs.at_least_one_output_enabled", + "outputs.papers_only_skips_proof_work", + "proof_runtime.no_lean_when_disabled", + "proof_runtime.no_smt_when_disabled", + ), + adapter="model", + actions=( + actions.start_autonomous_no_outputs, + actions.start_autonomous_papers_only, + actions.complete_topic_exploration, + actions.complete_brainstorm, + ), + must_exercise=("no_outputs_rejected", "papers_only"), + ), +) + + +def first_build_cross_field_coverage(test_file: str) -> tuple[InteractionCoverage, ...]: + return tuple(scenario.coverage_record(test_file) for scenario in CROSS_FIELD_SCENARIOS) diff --git a/tests/workflow_harness/exercise_evidence.py b/tests/workflow_harness/exercise_evidence.py new file mode 100644 index 0000000..67afe70 --- /dev/null +++ b/tests/workflow_harness/exercise_evidence.py @@ -0,0 +1,196 @@ +from __future__ import annotations + +from collections.abc import Iterable + +from .model import WorkflowModel + + +KNOWN_EXERCISE_TOKENS = frozenset( + { + "assistant_disable", + "assistant_pack", + "context_overflow", + "manual_archive", + "manual_aggregator_accept", + "manual_aggregator_clear", + "manual_proof", + "leanoj_clear", + "leanoj_draft", + "leanoj_force", + "leanoj_skip", + "paper_checkpoint", + "no_outputs_rejected", + "papers_only", + "prompt_context", + "rag_source_exclusion", + "mandatory_source_overflow", + "provider_pause", + "registered_proof_event", + "reset_credit", + "scoped_event", + "start_blocked", + "stop_resume", + } +) + + +def observed_exercise_tokens(model: WorkflowModel) -> frozenset[str]: + event_types = {event.event_type for event in model.events} + replay = model.replay + observed: set[str] = set() + + if model.provider.pause_count >= 1 and "provider_paused" in event_types: + observed.add("provider_pause") + stop_indexes = [index for index, action in enumerate(replay) if action == "stop()"] + resume_indexes = [ + index for index, action in enumerate(replay) if action.startswith("resume(") + ] + if ( + stop_indexes + and resume_indexes + and any(stop_index < resume_index for stop_index in stop_indexes for resume_index in resume_indexes) + and "stopped" in event_types + and "resumed" in event_types + ): + observed.add("stop_resume") + paused_event_indexes = [ + index + for index, event in enumerate(model.events) + if event.event_type == "provider_paused" + ] + resumed_event_indexes = [ + index + for index, event in enumerate(model.events) + if event.event_type == "provider_resumed" + ] + if ( + not model.provider.credit_exhausted + and paused_event_indexes + and resumed_event_indexes + and any( + paused_index < resumed_index + for paused_index in paused_event_indexes + for resumed_index in resumed_event_indexes + ) + and model.checkpoint.get("paused") is False + and model.checkpoint.get("phase") + ): + observed.add("reset_credit") + if any(action.startswith("refresh_assistant_pack(") for action in replay): + observed.add("assistant_pack") + if ( + any("toggle_session_history_memory(enabled=False)" in action for action in replay) + and not model.assistant.enabled + and not model.assistant.live_pack + ): + observed.add("assistant_disable") + if ( + any(action.startswith("prepare_prompt_context(") for action in replay) + and { + "user_prompt_direct_injected", + "validator_assistant_memory_excluded", + "proof_source_context_present", + "generated_appendices_stripped", + }.issubset(model.exercise_observations) + ): + observed.add("prompt_context") + if "manual-proof-1" in model.manual_proofs_active.union(model.manual_proofs_archived): + observed.add("manual_proof") + if "manual-proof-1" in model.manual_proofs_archived and not model.manual_proofs_active: + observed.add("manual_archive") + if model.manual_aggregator_submissions or model.manual_aggregator_history: + observed.add("manual_aggregator_accept") + if model.manual_aggregator_history and not model.manual_aggregator_submissions: + observed.add("manual_aggregator_clear") + if model.leanoj_draft_written and ( + model.leanoj_master_proof or model.leanoj_draft_preserved_on_resume or model.leanoj_cleared + ): + observed.add("leanoj_draft") + if model.leanoj_skip_count: + observed.add("leanoj_skip") + if model.leanoj_force_count: + observed.add("leanoj_force") + if model.leanoj_cleared and not model.leanoj_master_proof: + observed.add("leanoj_clear") + if model.autonomous_paper_checkpoint_count: + observed.add("paper_checkpoint") + + explicitly_scoped_events = [ + event + for event in model.events + if event.event_type in {"proof_progress", "proof_verified"} + ] + if explicitly_scoped_events and all( + "scope" in event.payload and "phase" in event.payload + for event in explicitly_scoped_events + ): + observed.add("scoped_event") + if "registered-proof-1" in model.autonomous_proofs and any( + event.event_type == "proof_verified" + and event.payload.get("proof_id") == "registered-proof-1" + for event in model.events + ): + observed.add("registered_proof_event") + if any( + event.event_type == "start_rejected" + and event.payload.get("reason") == "no_allowed_outputs" + for event in model.events + ): + observed.add("no_outputs_rejected") + if ( + any( + action.startswith("start_autonomous(") + and "papers=True" in action + and "proofs=False" in action + for action in replay + ) + and any(action.startswith("complete_brainstorm(") for action in replay) + and not model.allow_mathematical_proofs + and model.allow_research_papers + and model.lean.invocations == 0 + and model.smt.invocations == 0 + ): + observed.add("papers_only") + if "start_blocked" in event_types: + observed.add("start_blocked") + if ( + "context_overflow_error" in event_types + and "proof_context_overflow" in event_types + and model.terminal_stop_events == 1 + ): + observed.add("context_overflow") + if ( + any(action.startswith("verify_rag_source_exclusion(") for action in replay) + and "direct_sources_excluded_from_rag" in model.exercise_observations + ): + observed.add("rag_source_exclusion") + if ( + any(action.startswith("reject_mandatory_source_overflow(") for action in replay) + and "mandatory_source_overflow_rejected_visible" in model.exercise_observations + ): + observed.add("mandatory_source_overflow") + + return frozenset(observed) + + +def assert_exercise_tokens( + model: WorkflowModel, + required: Iterable[str], + *, + scenario_id: str, +) -> frozenset[str]: + required_tokens = frozenset(required) + unknown = required_tokens - KNOWN_EXERCISE_TOKENS + if unknown: + raise AssertionError( + f"{scenario_id} has unknown must_exercise tokens: {sorted(unknown)!r}." + ) + + observed = observed_exercise_tokens(model) + missing = required_tokens - observed + if missing: + raise AssertionError( + f"{scenario_id} did not exercise required behavior {sorted(missing)!r}; " + f"observed {sorted(observed)!r}." + ) + return observed diff --git a/tests/workflow_harness/invariant_catalog.py b/tests/workflow_harness/invariant_catalog.py new file mode 100644 index 0000000..941e3e4 --- /dev/null +++ b/tests/workflow_harness/invariant_catalog.py @@ -0,0 +1,685 @@ +from __future__ import annotations + +from collections.abc import Callable +from dataclasses import dataclass +from pathlib import Path +from typing import Literal, Protocol + +from .model import FakeAssistantMemory, FakeLean, FakeProvider, FakeSmt, WorkflowEvent, WorkflowMode, WorkflowPhase + + +AdapterType = Literal["model", "real_route", "real_coordinator", "browser_smoke"] + + +class InvariantState(Protocol): + runtime_root: Path + mode: WorkflowMode + phase: WorkflowPhase + allow_mathematical_proofs: bool + allow_research_papers: bool + provider: FakeProvider + lean: FakeLean + smt: FakeSmt + assistant: FakeAssistantMemory + autonomous_proofs: set[str] + manual_proofs_active: set[str] + manual_proofs_archived: set[str] + manual_proofs_before_clear: set[str] + active_owners: set[WorkflowMode] + background_owners: set[WorkflowMode] + pending_child_tasks: int + stale_child_outputs_fenced: bool + stale_child_outputs_accepted: int + truncation_failures: int + provider_pause_from_truncation: int + hosted_desktop_route_attempted: bool + hosted_desktop_route_unavailable: bool + hosted_route_attempts: set[str] + hosted_route_unavailable: set[str] + active_state_cleared: bool + history_preserved: bool + completed_brainstorm_generated_paper: bool + replayed_completed_brainstorm_handoff: bool + pruned_papers: set[str] + model_context_papers: set[str] + runtime_root_violations: int + resolved_runtime_paths: list[Path] + prompt_user_direct_injected: bool + validator_prompt_has_assistant_memory: bool + proof_source_context_required: bool + proof_source_context_present: bool + generated_appendices_stripped: bool + direct_sources_excluded_from_rag: bool + mandatory_source_overflow_visible: bool + exercise_observations: set[str] + persisted_events: list[WorkflowEvent] + terminal_stop_events: int + checkpoint: dict[str, object] + events: list[WorkflowEvent] + replay: list[str] + + +InvariantChecker = Callable[[InvariantState], None] + + +PAPER_PHASES = { + WorkflowPhase.PAPER_TITLE, + WorkflowPhase.PAPER_WRITING, + WorkflowPhase.PAPER_PROOF, +} + + +@dataclass(frozen=True) +class InvariantSpec: + invariant_id: str + field: str + crossed_fields: tuple[str, ...] + adapters: tuple[AdapterType, ...] + checker: InvariantChecker + description: str + + +def _fail(model: InvariantState, message: str) -> None: + replay = "\n".join(f"{index + 1}. {action}" for index, action in enumerate(model.replay)) + raise AssertionError(f"{message}\n\nReplay:\n{replay}") + + +def _assert_single_active_workflow(model: InvariantState) -> None: + owners = model.active_owners | model.background_owners + if model.mode is not WorkflowMode.NONE: + owners.add(model.mode) + if len(owners) > 1: + _fail(model, f"Competing workflow owners are active: {sorted(owner.value for owner in owners)!r}") + if model.mode is WorkflowMode.NONE and model.active_owners: + _fail(model, "Public mode is idle while a top-level owner remains active.") + if model.mode is not WorkflowMode.NONE and model.active_owners and model.active_owners != {model.mode}: + _fail(model, "Public mode and active workflow ownership disagree.") + + +def _assert_child_tasks_count_as_active(model: InvariantState) -> None: + if model.pending_child_tasks and not (model.active_owners | model.background_owners): + _fail(model, "Pending child workflow activity exists while no owner workflow is active.") + + +def _assert_parent_action_fences_child_outputs(model: InvariantState) -> None: + if not model.stale_child_outputs_fenced or model.stale_child_outputs_accepted: + _fail(model, "A parent workflow action did not fence stale child outputs.") + + +def _assert_lean_disabled_means_no_invocations(model: InvariantState) -> None: + if model.lean.blocked_invocations: + _fail(model, "Lean was invoked while lean4_enabled was false.") + + +def _assert_smt_disabled_means_no_invocations(model: InvariantState) -> None: + if model.smt.blocked_invocations: + _fail(model, "SMT was invoked while smt_enabled was false.") + + +def _assert_hosted_proof_settings_unavailable(model: InvariantState) -> None: + if model.hosted_desktop_route_attempted and not model.hosted_desktop_route_unavailable: + _fail(model, "Hosted mode attempted to run a desktop-only proof settings route.") + + +def _assert_truncation_is_attempt_failure(model: WorkflowModel) -> None: + if model.truncation_failures and model.provider_pause_from_truncation: + _fail(model, "Provider max-output truncation was treated as a provider pause.") + + +def _assert_at_least_one_output_enabled(model: WorkflowModel) -> None: + rejected = any( + event.event_type == "start_rejected" and event.payload.get("reason") == "no_allowed_outputs" + for event in model.events + ) + if not model.allow_mathematical_proofs and not model.allow_research_papers and not rejected: + _fail(model, "A workflow state disabled both proofs and papers without rejecting start.") + + +def _assert_proofs_only_never_enters_paper_phase(model: WorkflowModel) -> None: + if ( + model.mode is WorkflowMode.AUTONOMOUS + and model.allow_mathematical_proofs + and not model.allow_research_papers + and model.phase in PAPER_PHASES + ): + _fail(model, f"Proofs-only autonomous run entered paper phase {model.phase.value!r}.") + + +def _assert_papers_only_skips_proof_work(model: WorkflowModel) -> None: + if model.mode is WorkflowMode.AUTONOMOUS and not model.allow_mathematical_proofs: + if model.lean.invocations or model.smt.invocations: + _fail(model, "Papers-only autonomous run performed proof work.") + + +def _assert_provider_pause_preserves_checkpoint(model: InvariantState) -> None: + if model.phase is WorkflowPhase.PAUSED: + required = { + "paused", + "phase", + "resume_phase", + "source_type", + "source_id", + "trigger", + "candidate_cursor", + "proof_round", + } + missing = required - set(model.checkpoint) + if missing: + _fail(model, f"Provider pause checkpoint is missing {sorted(missing)!r}.") + if model.checkpoint.get("paused") is not True: + _fail(model, "Provider pause did not mark the checkpoint as paused.") + + +def _assert_stop_resume_preserves_pause(model: WorkflowModel) -> None: + if model.checkpoint.get("paused") and model.checkpoint.get("stopped"): + if not model.checkpoint.get("resume_phase"): + _fail(model, "Stop during provider pause did not preserve the resume phase.") + + +def _assert_reset_wakes_without_corrupting_checkpoint(model: WorkflowModel) -> None: + if model.provider.pause_count and model.checkpoint.get("paused") is False: + if not model.checkpoint.get("phase"): + _fail(model, "Provider reset cleared the checkpoint phase.") + + +def _assert_assistant_not_injected_into_validators(model: WorkflowModel) -> None: + if model.assistant.validator_injections: + _fail(model, "Assistant memory was injected into a validator role.") + + +def _assert_disabled_assistant_has_no_live_pack(model: WorkflowModel) -> None: + if not model.assistant.enabled and model.assistant.live_pack: + _fail(model, "Session History Memory is disabled but an Assistant pack remains live.") + + +def _assert_assistant_retrieval_non_blocking(model: WorkflowModel) -> None: + if model.assistant.blocked_parent_count: + _fail(model, "Assistant retrieval blocked parent workflow progress.") + + +def _assert_stagnant_pack_backoff_not_shutdown(model: WorkflowModel) -> None: + if model.assistant.stagnant_backoff_count and model.assistant.shutdown_count: + _fail(model, "Stagnant Assistant pack backoff shut down proof-memory retrieval.") + + +def _assert_manual_and_autonomous_proofs_are_isolated(model: WorkflowModel) -> None: + overlap = model.autonomous_proofs.intersection(model.manual_proofs_active) + if overlap: + _fail(model, f"Manual active proofs overlap autonomous proofs: {sorted(overlap)!r}") + + +def _assert_autonomous_events_not_manual(model: WorkflowModel) -> None: + leaked = [ + event + for event in model.events + if event.payload.get("scope") == "manual" and str(event.payload.get("proof_id", "")).startswith("auto") + ] + if leaked: + _fail(model, "Autonomous proof events leaked into manual proof scope.") + + +def _assert_clear_archives_manual_proofs(model: InvariantState) -> None: + if model.phase is WorkflowPhase.IDLE and model.mode is WorkflowMode.NONE and model.manual_proofs_active: + _fail(model, "Clear/idle state still has active manual proofs.") + missing = model.manual_proofs_before_clear - model.manual_proofs_archived + if missing: + _fail(model, f"Manual clear failed to archive active proofs: {sorted(missing)!r}") + + +def _assert_generated_appendices_stripped_from_prompts(model: WorkflowModel) -> None: + if not model.generated_appendices_stripped: + _fail(model, "Generated proof appendices were not stripped from model-visible prompts.") + + +def _assert_clear_removes_active_preserves_history(model: WorkflowModel) -> None: + if model.active_state_cleared and not model.history_preserved: + _fail(model, "Clear/reset removed preserved history.") + + +def _assert_completed_brainstorm_no_replay_handoff(model: WorkflowModel) -> None: + if model.completed_brainstorm_generated_paper and model.replayed_completed_brainstorm_handoff: + _fail(model, "Completed brainstorm with generated paper replayed proof/paper handoff.") + + +def _assert_pruned_papers_excluded_from_context(model: WorkflowModel) -> None: + overlap = model.pruned_papers.intersection(model.model_context_papers) + if overlap: + _fail(model, f"Pruned papers remain in model context: {sorted(overlap)!r}") + + +def _assert_runtime_roots_are_active_roots(model: InvariantState) -> None: + if model.runtime_root_violations: + _fail(model, "Runtime state resolved outside the active runtime root.") + for path in model.resolved_runtime_paths: + try: + path.resolve().relative_to(model.runtime_root.resolve()) + except ValueError: + _fail(model, f"Runtime path resolved outside active root: {path!s}") + + +def _assert_frontend_events_include_scope_phase(model: WorkflowModel) -> None: + checked_prefixes = ("proof_", "provider_", "assistant_", "workflow_") + for event in model.events: + if event.event_type.startswith(checked_prefixes): + if "scope" not in event.payload or "phase" not in event.payload: + _fail(model, f"Frontend event {event.event_type!r} lacks scope or phase.") + + +def _assert_context_overflow_route_identity(model: WorkflowModel) -> None: + required = {"workflow_mode", "role_id", "configured_model", "configured_provider"} + for event in model.events: + if event.event_type not in {"context_overflow_error", "proof_context_overflow"}: + continue + missing = required - set(event.payload) + if missing: + _fail(model, f"{event.event_type!r} lacks route identity fields {sorted(missing)!r}.") + has_effective_model = bool(event.payload.get("effective_model")) + has_effective_provider = bool(event.payload.get("effective_provider")) + if has_effective_model != has_effective_provider: + _fail(model, f"{event.event_type!r} has incomplete effective route identity.") + + +def _assert_context_overflow_persists_across_reload(model: WorkflowModel) -> None: + live = [event for event in model.events if event.event_type == "context_overflow_error"] + persisted = [ + event for event in model.persisted_events if event.event_type == "context_overflow_error" + ] + if live and not persisted: + _fail(model, "Fatal context overflow activity was not persisted for reload.") + if live and persisted and live[-1].payload != persisted[-1].payload: + _fail(model, "Reloaded context overflow activity lost live event metadata.") + + +def _assert_context_overflow_terminal_stop_once(model: WorkflowModel) -> None: + fatal_overflows = [ + event + for event in model.events + if event.event_type == "context_overflow_error" and event.payload.get("fatal") is not False + ] + if fatal_overflows and model.terminal_stop_events != 1: + _fail(model, "Fatal context overflow did not produce exactly one terminal stop activity.") + + +def _assert_proof_context_overflow_nonfatal(model: WorkflowModel) -> None: + proof_overflows = [ + event for event in model.events if event.event_type == "proof_context_overflow" + ] + if any(event.payload.get("fatal") is not False for event in proof_overflows): + _fail(model, "Per-proof context overflow was treated as fatal.") + + +def _assert_proof_verified_after_registration(model: WorkflowModel) -> None: + for event in model.events: + if event.event_type == "proof_verified": + proof_id = str(event.payload.get("proof_id") or "") + if not proof_id: + _fail(model, "proof_verified event did not include proof_id.") + if ( + proof_id not in model.autonomous_proofs + and proof_id not in model.manual_proofs_active + and proof_id not in model.manual_proofs_archived + ): + _fail(model, f"proof_verified emitted before proof registration: {proof_id!r}.") + + +def _assert_hosted_desktop_only_routes_unavailable(model: InvariantState) -> None: + required = {"proof_settings", "update", "pdf", "desktop_oauth"} + attempted = required & model.hosted_route_attempts + unavailable = required & model.hosted_route_unavailable + if attempted != unavailable: + _fail(model, f"Hosted desktop-only routes were not uniformly unavailable: {sorted(attempted - unavailable)!r}") + + +def _assert_user_prompt_direct_injected(model: WorkflowModel) -> None: + if not model.prompt_user_direct_injected: + _fail(model, "User prompt was not direct injected.") + + +def _assert_validator_excludes_assistant_memory(model: WorkflowModel) -> None: + if model.validator_prompt_has_assistant_memory: + _fail(model, "Validator prompt included Assistant memory.") + + +def _assert_proof_source_context_required(model: WorkflowModel) -> None: + if model.proof_source_context_required and not model.proof_source_context_present: + _fail(model, "Proof source context was required but absent.") + + +def _assert_direct_sources_excluded_from_rag(model: WorkflowModel) -> None: + if not model.direct_sources_excluded_from_rag: + _fail(model, "A directly injected source was duplicated into supplemental RAG evidence.") + + +def _assert_mandatory_source_overflow_visible(model: WorkflowModel) -> None: + if not model.mandatory_source_overflow_visible: + _fail(model, "Mandatory source overflow was truncated or hidden by optional RAG context.") + + +INVARIANT_CATALOG: tuple[InvariantSpec, ...] = ( + InvariantSpec( + invariant_id="runtime.single_active_workflow", + field="runtime_exclusivity", + crossed_fields=("provider_pause_resume", "workflow_filesystem_state", "websocket_api_contracts"), + adapters=("model", "real_route", "browser_smoke"), + checker=_assert_single_active_workflow, + description="Only one top-level workflow mode may be active at a time.", + ), + InvariantSpec( + invariant_id="runtime.child_tasks_count_as_active", + field="runtime_exclusivity", + crossed_fields=("provider_pause_resume", "workflow_filesystem_state", "websocket_api_contracts"), + adapters=("model", "real_route"), + checker=_assert_child_tasks_count_as_active, + description="Pending or background child activity counts as workflow ownership.", + ), + InvariantSpec( + invariant_id="runtime.parent_action_fences_child_outputs", + field="runtime_exclusivity", + crossed_fields=("workflow_filesystem_state", "websocket_api_contracts"), + adapters=("model",), + checker=_assert_parent_action_fences_child_outputs, + description="Parent actions fence stale lower-tier child outputs.", + ), + InvariantSpec( + invariant_id="proof_runtime.no_lean_when_disabled", + field="proof_runtime_gating", + crossed_fields=("allowed_outputs", "provider_pause_resume", "prompt_context"), + adapters=("model", "real_coordinator"), + checker=_assert_lean_disabled_means_no_invocations, + description="Lean proof work must not run when Lean is disabled.", + ), + InvariantSpec( + invariant_id="proof_runtime.no_smt_when_disabled", + field="proof_runtime_gating", + crossed_fields=("allowed_outputs", "prompt_context"), + adapters=("model",), + checker=_assert_smt_disabled_means_no_invocations, + description="SMT hint generation must not run when SMT is disabled.", + ), + InvariantSpec( + invariant_id="proof_runtime.hosted_proof_settings_unavailable", + field="proof_runtime_gating", + crossed_fields=("websocket_api_contracts", "allowed_outputs"), + adapters=("model", "real_route"), + checker=_assert_hosted_proof_settings_unavailable, + description="Hosted/generic proof settings routes remain unavailable.", + ), + InvariantSpec( + invariant_id="proof_runtime.truncation_is_attempt_failure", + field="proof_runtime_gating", + crossed_fields=("provider_pause_resume", "prompt_context"), + adapters=("model",), + checker=_assert_truncation_is_attempt_failure, + description="Provider max-output truncation is a proof attempt failure, not a provider pause.", + ), + InvariantSpec( + invariant_id="outputs.at_least_one_output_enabled", + field="allowed_outputs", + crossed_fields=("proof_runtime_gating",), + adapters=("model", "real_route", "browser_smoke"), + checker=_assert_at_least_one_output_enabled, + description="Workflow starts reject requests with both proof and paper outputs disabled.", + ), + InvariantSpec( + invariant_id="outputs.proofs_only_no_paper_phase", + field="allowed_outputs", + crossed_fields=("proof_runtime_gating", "provider_pause_resume", "workflow_filesystem_state"), + adapters=("model", "real_coordinator"), + checker=_assert_proofs_only_never_enters_paper_phase, + description="Proofs-only autonomous runs must not enter title, paper writing, or paper proof phases.", + ), + InvariantSpec( + invariant_id="outputs.papers_only_skips_proof_work", + field="allowed_outputs", + crossed_fields=("proof_runtime_gating", "provider_pause_resume"), + adapters=("model",), + checker=_assert_papers_only_skips_proof_work, + description="Papers-only autonomous runs skip proof model work and proof tool execution.", + ), + InvariantSpec( + invariant_id="provider.pause_preserves_checkpoint", + field="provider_pause_resume", + crossed_fields=("workflow_filesystem_state", "proof_runtime_gating", "runtime_exclusivity"), + adapters=("model", "real_coordinator"), + checker=_assert_provider_pause_preserves_checkpoint, + description="Provider-credit pauses preserve a resumable checkpoint.", + ), + InvariantSpec( + invariant_id="provider.stop_resume_preserves_pause", + field="provider_pause_resume", + crossed_fields=("workflow_filesystem_state", "runtime_exclusivity"), + adapters=("model",), + checker=_assert_stop_resume_preserves_pause, + description="Stop/resume during provider pause preserves pause checkpoint state.", + ), + InvariantSpec( + invariant_id="provider.reset_wakes_without_corrupting_checkpoint", + field="provider_pause_resume", + crossed_fields=("workflow_filesystem_state", "proof_runtime_gating"), + adapters=("model",), + checker=_assert_reset_wakes_without_corrupting_checkpoint, + description="Provider reset wakes paused work without corrupting checkpoint recovery.", + ), + InvariantSpec( + invariant_id="assistant.no_validator_injection", + field="assistant_memory", + crossed_fields=("prompt_context", "proof_scope_isolation", "provider_pause_resume"), + adapters=("model",), + checker=_assert_assistant_not_injected_into_validators, + description="Validators must not receive Assistant proof-memory packs.", + ), + InvariantSpec( + invariant_id="assistant.disable_clears_live_pack", + field="assistant_memory", + crossed_fields=("prompt_context", "proof_scope_isolation"), + adapters=("model",), + checker=_assert_disabled_assistant_has_no_live_pack, + description="Disabling Session History Memory clears the live Assistant pack.", + ), + InvariantSpec( + invariant_id="assistant.non_blocking_retrieval", + field="assistant_memory", + crossed_fields=("prompt_context", "provider_pause_resume"), + adapters=("model",), + checker=_assert_assistant_retrieval_non_blocking, + description="Assistant retrieval is optional and never blocks parent workflow progress.", + ), + InvariantSpec( + invariant_id="assistant.stagnant_pack_backoff_not_shutdown", + field="assistant_memory", + crossed_fields=("provider_pause_resume",), + adapters=("model",), + checker=_assert_stagnant_pack_backoff_not_shutdown, + description="Stagnant same-pack retrieval backs off without shutting down the run.", + ), + InvariantSpec( + invariant_id="proof_scope.manual_not_in_autonomous_current", + field="proof_scope_isolation", + crossed_fields=("assistant_memory", "workflow_filesystem_state", "websocket_api_contracts"), + adapters=("model", "real_route", "real_coordinator"), + checker=_assert_manual_and_autonomous_proofs_are_isolated, + description="Manual active proofs must not overlap autonomous current-session proofs.", + ), + InvariantSpec( + invariant_id="proof_scope.autonomous_events_not_manual", + field="proof_scope_isolation", + crossed_fields=("websocket_api_contracts", "workflow_filesystem_state"), + adapters=("model", "real_coordinator"), + checker=_assert_autonomous_events_not_manual, + description="Autonomous proof events do not populate manual proof scopes.", + ), + InvariantSpec( + invariant_id="proof_scope.manual_clear_archives_active", + field="proof_scope_isolation", + crossed_fields=("workflow_filesystem_state", "assistant_memory"), + adapters=("model", "real_route", "real_coordinator"), + checker=_assert_clear_archives_manual_proofs, + description="Manual clear leaves no active manual proofs in an idle cleared state.", + ), + InvariantSpec( + invariant_id="proof_scope.generated_appendices_stripped_from_prompts", + field="proof_scope_isolation", + crossed_fields=("prompt_context", "workflow_filesystem_state"), + adapters=("model", "real_coordinator"), + checker=_assert_generated_appendices_stripped_from_prompts, + description="Generated proof appendices are stripped from future model-visible source prompts.", + ), + InvariantSpec( + invariant_id="state.clear_removes_active_preserves_history", + field="workflow_filesystem_state", + crossed_fields=("proof_scope_isolation", "assistant_memory"), + adapters=("model", "real_route", "real_coordinator"), + checker=_assert_clear_removes_active_preserves_history, + description="Clear removes active runtime state while preserving intended history archives.", + ), + InvariantSpec( + invariant_id="state.completed_brainstorm_no_replay_handoff", + field="workflow_filesystem_state", + crossed_fields=("allowed_outputs", "provider_pause_resume"), + adapters=("model",), + checker=_assert_completed_brainstorm_no_replay_handoff, + description="Completed brainstorms with generated papers do not replay proof/paper handoff on resume.", + ), + InvariantSpec( + invariant_id="state.pruned_papers_excluded_from_context", + field="workflow_filesystem_state", + crossed_fields=("prompt_context",), + adapters=("model", "real_route"), + checker=_assert_pruned_papers_excluded_from_context, + description="Pruned papers remain history but are excluded from model context.", + ), + InvariantSpec( + invariant_id="state.runtime_roots_are_active_roots", + field="workflow_filesystem_state", + crossed_fields=("runtime_exclusivity",), + adapters=("model", "real_coordinator"), + checker=_assert_runtime_roots_are_active_roots, + description="Runtime files resolve under active runtime roots.", + ), + InvariantSpec( + invariant_id="events.frontend_events_include_scope_phase", + field="websocket_api_contracts", + crossed_fields=("proof_scope_isolation", "provider_pause_resume"), + adapters=("model",), + checker=_assert_frontend_events_include_scope_phase, + description="Frontend-consumed proof/provider/Assistant/workflow events include scope and phase.", + ), + InvariantSpec( + invariant_id="events.context_overflow_route_identity", + field="websocket_api_contracts", + crossed_fields=("provider_pause_resume", "workflow_filesystem_state"), + adapters=("model", "real_coordinator", "browser_smoke"), + checker=_assert_context_overflow_route_identity, + description="Context-overflow activity identifies configured and effective model routes.", + ), + InvariantSpec( + invariant_id="events.context_overflow_persists_across_reload", + field="websocket_api_contracts", + crossed_fields=("workflow_filesystem_state",), + adapters=("model", "real_coordinator", "browser_smoke"), + checker=_assert_context_overflow_persists_across_reload, + description="Fatal context-overflow metadata survives the workflow's persistence and reload path.", + ), + InvariantSpec( + invariant_id="events.context_overflow_terminal_stop_once", + field="websocket_api_contracts", + crossed_fields=("runtime_exclusivity", "workflow_filesystem_state"), + adapters=("model", "real_coordinator", "browser_smoke"), + checker=_assert_context_overflow_terminal_stop_once, + description="A fatal context overflow produces one terminal workflow-stop activity.", + ), + InvariantSpec( + invariant_id="events.proof_context_overflow_nonfatal", + field="websocket_api_contracts", + crossed_fields=("proof_runtime_gating", "provider_pause_resume"), + adapters=("model", "real_coordinator", "browser_smoke"), + checker=_assert_proof_context_overflow_nonfatal, + description="Per-proof context overflow is scoped and nonfatal to its parent workflow.", + ), + InvariantSpec( + invariant_id="events.proof_verified_after_registration", + field="websocket_api_contracts", + crossed_fields=("proof_scope_isolation",), + adapters=("model",), + checker=_assert_proof_verified_after_registration, + description="proof_verified emits only after proof registration or reuse and includes proof_id.", + ), + InvariantSpec( + invariant_id="api.hosted_desktop_only_routes_unavailable", + field="websocket_api_contracts", + crossed_fields=("proof_runtime_gating", "runtime_exclusivity"), + adapters=("model", "browser_smoke"), + checker=_assert_hosted_desktop_only_routes_unavailable, + description="Hosted/generic routes report desktop-only behavior unavailable.", + ), + InvariantSpec( + invariant_id="prompt.user_prompt_direct_injected", + field="prompt_context", + crossed_fields=("assistant_memory", "proof_runtime_gating"), + adapters=("model",), + checker=_assert_user_prompt_direct_injected, + description="User prompt remains direct injected where required.", + ), + InvariantSpec( + invariant_id="prompt.validator_excludes_assistant_memory", + field="prompt_context", + crossed_fields=("assistant_memory",), + adapters=("model",), + checker=_assert_validator_excludes_assistant_memory, + description="Validator prompts exclude Assistant proof-memory packs.", + ), + InvariantSpec( + invariant_id="prompt.proof_source_context_required", + field="prompt_context", + crossed_fields=("proof_runtime_gating",), + adapters=("model",), + checker=_assert_proof_source_context_required, + description="Proof formalization receives mandatory source context or fails visibly.", + ), + InvariantSpec( + invariant_id="prompt.direct_sources_excluded_from_rag", + field="prompt_context", + crossed_fields=("workflow_filesystem_state",), + adapters=("model", "real_coordinator"), + checker=_assert_direct_sources_excluded_from_rag, + description="Directly injected sources are excluded from supplemental RAG evidence.", + ), + InvariantSpec( + invariant_id="prompt.mandatory_source_overflow_fails_visible", + field="prompt_context", + crossed_fields=("proof_runtime_gating", "websocket_api_contracts"), + adapters=("model", "real_coordinator"), + checker=_assert_mandatory_source_overflow_visible, + description="Mandatory source overflow fails visibly before model calls or RAG substitution.", + ), + InvariantSpec( + invariant_id="prompt.generated_appendices_stripped", + field="prompt_context", + crossed_fields=("proof_scope_isolation",), + adapters=("model", "real_coordinator"), + checker=_assert_generated_appendices_stripped_from_prompts, + description="Model-visible paper and brainstorm source reads strip generated proof appendices.", + ), +) + + +INVARIANTS_BY_ID = {spec.invariant_id: spec for spec in INVARIANT_CATALOG} +FIRST_BUILD_INVARIANT_IDS = frozenset(INVARIANTS_BY_ID) +FIRST_BUILD_FIELDS = frozenset(spec.field for spec in INVARIANT_CATALOG) + + +def get_invariant(invariant_id: str) -> InvariantSpec: + return INVARIANTS_BY_ID[invariant_id] + + +def assert_invariant(model: InvariantState, invariant_id: str) -> None: + get_invariant(invariant_id).checker(model) + + +def assert_all_catalog_invariants(model: InvariantState) -> None: + for spec in INVARIANT_CATALOG: + spec.checker(model) + + +def invariant_ids_for_adapter(adapter: AdapterType) -> set[str]: + return {spec.invariant_id for spec in INVARIANT_CATALOG if adapter in spec.adapters} diff --git a/tests/workflow_harness/invariants.py b/tests/workflow_harness/invariants.py new file mode 100644 index 0000000..d810776 --- /dev/null +++ b/tests/workflow_harness/invariants.py @@ -0,0 +1,49 @@ +from __future__ import annotations + +from .invariant_catalog import assert_all_catalog_invariants, assert_invariant +from .model import WorkflowModel + + +def assert_observed_invariant(model: WorkflowModel, invariant_id: str) -> None: + """Assert one real-adapter invariant only after the test marks it observed.""" + observed = getattr(model, "observed_invariants", None) + if observed is not None and invariant_id not in observed: + raise AssertionError(f"Invariant {invariant_id!r} was not exercised by this observation.") + assert_invariant(model, invariant_id) + + +def assert_single_active_workflow(model: WorkflowModel) -> None: + assert_invariant(model, "runtime.single_active_workflow") + + +def assert_lean_disabled_means_no_invocations(model: WorkflowModel) -> None: + assert_invariant(model, "proof_runtime.no_lean_when_disabled") + + +def assert_proofs_only_never_enters_paper_phase(model: WorkflowModel) -> None: + assert_invariant(model, "outputs.proofs_only_no_paper_phase") + + +def assert_provider_pause_preserves_checkpoint(model: WorkflowModel) -> None: + assert_invariant(model, "provider.pause_preserves_checkpoint") + + +def assert_assistant_not_injected_into_validators(model: WorkflowModel) -> None: + assert_invariant(model, "assistant.no_validator_injection") + + +def assert_disabled_assistant_has_no_live_pack(model: WorkflowModel) -> None: + assert_invariant(model, "assistant.disable_clears_live_pack") + + +def assert_manual_and_autonomous_proofs_are_isolated(model: WorkflowModel) -> None: + assert_invariant(model, "proof_scope.manual_not_in_autonomous_current") + + +def assert_clear_archives_manual_proofs(model: WorkflowModel) -> None: + assert_invariant(model, "proof_scope.manual_clear_archives_active") + + +def assert_all_invariants(model: WorkflowModel) -> None: + assert_all_catalog_invariants(model) + diff --git a/tests/workflow_harness/model.py b/tests/workflow_harness/model.py new file mode 100644 index 0000000..ab716fc --- /dev/null +++ b/tests/workflow_harness/model.py @@ -0,0 +1,622 @@ +from __future__ import annotations + +from dataclasses import dataclass, field +from enum import Enum +from pathlib import Path + + +class WorkflowMode(str, Enum): + NONE = "none" + AUTONOMOUS = "autonomous" + MANUAL_COMPILER = "manual_compiler" + MANUAL_AGGREGATOR = "manual_aggregator" + LEANOJ = "leanoj" + + +class WorkflowPhase(str, Enum): + IDLE = "idle" + TOPIC_EXPLORATION = "topic_exploration" + TIER1_AGGREGATION = "tier1_aggregation" + BRAINSTORM_PROOF = "brainstorm_proof_verification" + PAPER_TITLE = "paper_title_selection" + PAPER_WRITING = "paper_writing" + PAPER_PROOF = "paper_proof_verification" + TIER3 = "tier3_final_answer" + MANUAL_PROOF = "manual_proof_check" + MANUAL_AGGREGATION = "manual_aggregation" + LEANOJ_BRAINSTORM = "leanoj_brainstorm" + LEANOJ_FINAL = "leanoj_final" + PAUSED = "provider_credit_pause" + + +@dataclass(frozen=True) +class WorkflowEvent: + event_type: str + payload: dict[str, object] = field(default_factory=dict) + + +@dataclass +class FakeProvider: + credit_exhausted: bool = False + pause_count: int = 0 + + def exhaust_credit(self) -> None: + self.credit_exhausted = True + self.pause_count += 1 + + def reset_credit(self) -> None: + self.credit_exhausted = False + + +@dataclass +class FakeLean: + enabled: bool = False + requests: int = 0 + invocations: int = 0 + blocked_invocations: int = 0 + + def run(self) -> bool: + self.requests += 1 + if not self.enabled: + self.blocked_invocations += 1 + return False + self.invocations += 1 + return True + + +@dataclass +class FakeSmt: + enabled: bool = False + requests: int = 0 + invocations: int = 0 + blocked_invocations: int = 0 + + def run(self) -> bool: + self.requests += 1 + if not self.enabled: + self.blocked_invocations += 1 + return False + self.invocations += 1 + return True + + +@dataclass +class FakeAssistantMemory: + enabled: bool = True + live_pack: tuple[str, ...] = () + validator_injections: int = 0 + retrieval_count: int = 0 + blocked_parent_count: int = 0 + stagnant_backoff_count: int = 0 + shutdown_count: int = 0 + + def set_enabled(self, enabled: bool) -> None: + self.enabled = enabled + if not enabled: + self.live_pack = () + + def refresh_pack(self, *proof_ids: str) -> None: + if self.enabled: + self.live_pack = tuple(proof_ids[:7]) + + def inject_into_validator(self) -> None: + self.validator_injections += 1 + + def retrieve_non_blocking(self, *proof_ids: str) -> None: + self.retrieval_count += 1 + self.refresh_pack(*proof_ids) + + def mark_stagnant_backoff(self) -> None: + self.stagnant_backoff_count += 1 + + +@dataclass +class WorkflowModel: + runtime_root: Path + mode: WorkflowMode = WorkflowMode.NONE + phase: WorkflowPhase = WorkflowPhase.IDLE + previous_phase: WorkflowPhase = WorkflowPhase.IDLE + allow_mathematical_proofs: bool = True + allow_research_papers: bool = True + provider: FakeProvider = field(default_factory=FakeProvider) + lean: FakeLean = field(default_factory=FakeLean) + smt: FakeSmt = field(default_factory=FakeSmt) + assistant: FakeAssistantMemory = field(default_factory=FakeAssistantMemory) + autonomous_proofs: set[str] = field(default_factory=set) + manual_proofs_active: set[str] = field(default_factory=set) + manual_proofs_archived: set[str] = field(default_factory=set) + manual_proofs_before_clear: set[str] = field(default_factory=set) + manual_aggregator_submissions: list[str] = field(default_factory=list) + manual_aggregator_history: list[str] = field(default_factory=list) + leanoj_master_proof: str = "" + leanoj_draft_written: bool = False + leanoj_draft_preserved_on_resume: bool = False + leanoj_cleared: bool = False + leanoj_skip_count: int = 0 + leanoj_force_count: int = 0 + autonomous_paper_checkpoint_count: int = 0 + active_owners: set[WorkflowMode] = field(default_factory=set) + background_owners: set[WorkflowMode] = field(default_factory=set) + pending_child_tasks: int = 0 + stale_child_outputs_fenced: bool = True + parent_generation: int = 0 + child_task_generations: list[int] = field(default_factory=list) + stale_child_outputs_accepted: int = 0 + truncation_failures: int = 0 + provider_pause_from_truncation: int = 0 + hosted_desktop_route_attempted: bool = False + hosted_desktop_route_unavailable: bool = True + hosted_route_attempts: set[str] = field(default_factory=set) + hosted_route_unavailable: set[str] = field(default_factory=set) + active_state_cleared: bool = False + history_preserved: bool = True + completed_brainstorm_generated_paper: bool = False + replayed_completed_brainstorm_handoff: bool = False + pruned_papers: set[str] = field(default_factory=set) + model_context_papers: set[str] = field(default_factory=set) + runtime_root_violations: int = 0 + resolved_runtime_paths: list[Path] = field(default_factory=list) + prompt_user_direct_injected: bool = True + validator_prompt_has_assistant_memory: bool = False + proof_source_context_required: bool = True + proof_source_context_present: bool = True + generated_appendices_stripped: bool = True + direct_sources_excluded_from_rag: bool = True + mandatory_source_overflow_visible: bool = True + exercise_observations: set[str] = field(default_factory=set) + persisted_events: list[WorkflowEvent] = field(default_factory=list) + terminal_stop_events: int = 0 + checkpoint: dict[str, object] = field(default_factory=dict) + events: list[WorkflowEvent] = field(default_factory=list) + replay: list[str] = field(default_factory=list) + + def record(self, action: str, **payload: object) -> None: + if payload: + details = ", ".join(f"{key}={value!r}" for key, value in sorted(payload.items())) + self.replay.append(f"{action}({details})") + else: + self.replay.append(f"{action}()") + + def emit(self, event_type: str, **payload: object) -> None: + self.events.append(WorkflowEvent(event_type=event_type, payload=payload)) + + def start_autonomous( + self, + *, + proofs: bool = True, + papers: bool = True, + lean_enabled: bool = False, + ) -> None: + self.record("start_autonomous", proofs=proofs, papers=papers, lean_enabled=lean_enabled) + if self.mode is not WorkflowMode.NONE: + self.emit("start_blocked", active_mode=self.mode.value) + return + if not proofs and not papers: + self.emit("start_rejected", reason="no_allowed_outputs") + return + + self.mode = WorkflowMode.AUTONOMOUS + self.active_owners.add(self.mode) + self.phase = WorkflowPhase.TOPIC_EXPLORATION + self.allow_mathematical_proofs = proofs + self.allow_research_papers = papers + self.lean.enabled = lean_enabled + self.lean.invocations = 0 + self.lean.blocked_invocations = 0 + self.lean.requests = 0 + self.smt.invocations = 0 + self.smt.blocked_invocations = 0 + self.smt.requests = 0 + self.checkpoint = {"phase": self.phase.value, "mode": self.mode.value} + self.emit("started", mode=self.mode.value, phase=self.phase.value) + + def start_autonomous_with_no_outputs(self) -> None: + self.record("start_autonomous_with_no_outputs") + self.start_autonomous(proofs=False, papers=False, lean_enabled=False) + + def start_manual_compiler(self) -> None: + self.record("start_manual_compiler") + if self.mode is not WorkflowMode.NONE: + self.emit("start_blocked", active_mode=self.mode.value) + return + self.mode = WorkflowMode.MANUAL_COMPILER + self.active_owners.add(self.mode) + self.phase = WorkflowPhase.PAPER_WRITING + self.lean.invocations = 0 + self.lean.blocked_invocations = 0 + self.lean.requests = 0 + self.smt.invocations = 0 + self.smt.blocked_invocations = 0 + self.smt.requests = 0 + self.emit("started", mode=self.mode.value, phase=self.phase.value) + + def start_manual_aggregator(self) -> None: + self.record("start_manual_aggregator") + if self.mode is not WorkflowMode.NONE: + self.emit("start_blocked", active_mode=self.mode.value) + return + self.mode = WorkflowMode.MANUAL_AGGREGATOR + self.active_owners.add(self.mode) + self.phase = WorkflowPhase.MANUAL_AGGREGATION + self.checkpoint = {"mode": self.mode.value, "phase": self.phase.value} + self.emit("started", mode=self.mode.value, phase=self.phase.value) + + def accept_manual_aggregator_submission(self) -> None: + self.record("accept_manual_aggregator_submission") + if self.mode is WorkflowMode.MANUAL_AGGREGATOR: + submission = f"submission-{len(self.manual_aggregator_submissions) + 1}" + self.manual_aggregator_submissions.append(submission) + self.checkpoint["submission_count"] = len(self.manual_aggregator_submissions) + + def start_leanoj(self) -> None: + self.record("start_leanoj") + if self.mode is not WorkflowMode.NONE: + self.emit("start_blocked", active_mode=self.mode.value) + return + self.mode = WorkflowMode.LEANOJ + self.active_owners.add(self.mode) + self.phase = WorkflowPhase.LEANOJ_BRAINSTORM + self.checkpoint = {"mode": self.mode.value, "phase": self.phase.value} + self.emit("started", mode=self.mode.value, phase=self.phase.value) + + def edit_leanoj_master_proof(self) -> None: + self.record("edit_leanoj_master_proof") + if self.mode is WorkflowMode.LEANOJ: + self.leanoj_master_proof = "theorem durable_draft : True := by\n trivial\n" + self.leanoj_draft_written = True + self.checkpoint["master_proof"] = self.leanoj_master_proof + + def skip_leanoj_brainstorm(self) -> None: + self.record("skip_leanoj_brainstorm") + if self.mode is WorkflowMode.LEANOJ: + self.leanoj_skip_count += 1 + self.phase = WorkflowPhase.LEANOJ_FINAL + self.checkpoint["phase"] = self.phase.value + + def force_leanoj_brainstorm(self) -> None: + self.record("force_leanoj_brainstorm") + if self.mode is WorkflowMode.LEANOJ: + self.leanoj_force_count += 1 + self.phase = WorkflowPhase.LEANOJ_BRAINSTORM + self.checkpoint["phase"] = self.phase.value + + def enter_autonomous_paper_checkpoint(self) -> None: + self.record("enter_autonomous_paper_checkpoint") + if ( + self.mode is WorkflowMode.AUTONOMOUS + and self.phase is WorkflowPhase.PAPER_TITLE + and self.allow_research_papers + ): + self.phase = WorkflowPhase.PAPER_PROOF + self.checkpoint.update( + { + "phase": self.phase.value, + "source_type": "paper", + "source_id": "paper-1", + "trigger": "post_paper", + } + ) + + def complete_autonomous_paper_checkpoint(self) -> None: + self.record("complete_autonomous_paper_checkpoint") + if self.mode is WorkflowMode.AUTONOMOUS and self.phase is WorkflowPhase.PAPER_PROOF: + if self.allow_mathematical_proofs and self.lean.enabled: + self.lean.run() + self.autonomous_proofs.add("paper-proof-1") + self.emit( + "proof_verified", + proof_id="paper-proof-1", + scope="autonomous", + phase=self.phase.value, + ) + self.autonomous_paper_checkpoint_count += 1 + self.phase = WorkflowPhase.TOPIC_EXPLORATION + self.checkpoint["phase"] = self.phase.value + + def complete_topic_exploration(self) -> None: + self.record("complete_topic_exploration") + if self.mode is WorkflowMode.AUTONOMOUS and self.phase is WorkflowPhase.TOPIC_EXPLORATION: + self.phase = WorkflowPhase.TIER1_AGGREGATION + self.checkpoint["phase"] = self.phase.value + self.emit("phase_changed", phase=self.phase.value) + + def start_child_task(self) -> None: + self.record("start_child_task") + self.pending_child_tasks += 1 + if self.mode is not WorkflowMode.NONE: + self.background_owners.add(self.mode) + self.child_task_generations.append(self.parent_generation) + + def stale_child_output_arrives_after_parent_action(self) -> None: + self.record("stale_child_output_arrives_after_parent_action") + child_generation = self.child_task_generations[0] if self.child_task_generations else -1 + fenced = child_generation != self.parent_generation + self.stale_child_outputs_fenced = fenced + if not fenced: + self.stale_child_outputs_accepted += 1 + + def complete_brainstorm(self) -> None: + self.record("complete_brainstorm") + if self.mode is not WorkflowMode.AUTONOMOUS: + return + if self.provider.credit_exhausted and self.allow_mathematical_proofs: + self.previous_phase = self.phase + self.phase = WorkflowPhase.PAUSED + self.checkpoint = { + "phase": self.phase.value, + "resume_phase": self.previous_phase.value, + "mode": self.mode.value, + "paused": True, + "source_type": "brainstorm", + "source_id": "brainstorm-1", + "trigger": "post_brainstorm", + "candidate_cursor": 0, + "proof_round": 1, + } + self.emit( + "provider_paused", + reason="openrouter_credit_exhaustion", + scope="autonomous", + phase=self.phase.value, + ) + return + if self.allow_mathematical_proofs and self.lean.enabled: + self.lean.run() + self.autonomous_proofs.add("auto-proof-1") + self.phase = WorkflowPhase.BRAINSTORM_PROOF + self.checkpoint["proof_round_complete"] = True + self.emit("proof_verified", proof_id="auto-proof-1", scope="autonomous", phase=self.phase.value) + if self.allow_research_papers: + self.phase = WorkflowPhase.PAPER_TITLE + else: + self.phase = WorkflowPhase.TOPIC_EXPLORATION + self.checkpoint["phase"] = self.phase.value + self.emit("phase_changed", phase=self.phase.value) + + def run_smt_hint_generation(self) -> None: + self.record("run_smt_hint_generation") + self.exercise_observations.add("smt_gate_exercised") + if self.smt.enabled: + self.smt.run() + + def simulate_provider_output_truncation(self) -> None: + self.record("simulate_provider_output_truncation") + self.truncation_failures += 1 + self.emit( + "proof_attempt_failed", + reason="provider_output_truncation", + scope="autonomous", + phase=self.phase.value, + ) + + def try_hosted_desktop_only_route(self) -> None: + self.record("try_hosted_desktop_only_route") + self.hosted_desktop_route_attempted = True + self.hosted_desktop_route_unavailable = True + route_categories = {"proof_settings", "update", "pdf", "desktop_oauth"} + self.hosted_route_attempts.update(route_categories) + self.hosted_route_unavailable.update(route_categories) + self.emit("route_unavailable", deployment="hosted", reason="desktop_only") + + def attempt_disabled_lean_checkpoint(self) -> None: + self.record("attempt_disabled_lean_checkpoint") + self.exercise_observations.add("lean_gate_exercised") + if self.lean.enabled: + self.lean.run() + + def complete_brainstorm_with_generated_paper(self) -> None: + self.record("complete_brainstorm_with_generated_paper") + self.completed_brainstorm_generated_paper = True + self.checkpoint["completed_brainstorm_generated_paper"] = True + + def resume_completed_brainstorm(self) -> None: + self.record("resume_completed_brainstorm") + if self.completed_brainstorm_generated_paper: + self.replayed_completed_brainstorm_handoff = False + + def prune_paper(self, paper_id: str = "paper-1") -> None: + self.record("prune_paper", paper_id=paper_id) + self.pruned_papers.add(paper_id) + self.model_context_papers.discard(paper_id) + + def add_paper_to_model_context(self, paper_id: str = "paper-1") -> None: + self.record("add_paper_to_model_context", paper_id=paper_id) + self.model_context_papers.add(paper_id) + + def prepare_prompt_context(self) -> None: + self.record("prepare_prompt_context") + self.prompt_user_direct_injected = True + self.validator_prompt_has_assistant_memory = False + self.proof_source_context_required = True + self.proof_source_context_present = True + self.generated_appendices_stripped = True + self.exercise_observations.update( + { + "user_prompt_direct_injected", + "validator_assistant_memory_excluded", + "proof_source_context_present", + "generated_appendices_stripped", + } + ) + + def prepare_validator_prompt_with_live_assistant(self) -> None: + self.record("prepare_validator_prompt_with_live_assistant") + self.exercise_observations.add("validator_live_assistant_exclusion_exercised") + self.validator_prompt_has_assistant_memory = False + + def verify_rag_source_exclusion(self) -> None: + self.record("verify_rag_source_exclusion") + self.direct_sources_excluded_from_rag = True + self.exercise_observations.add("direct_sources_excluded_from_rag") + + def reject_mandatory_source_overflow(self) -> None: + self.record("reject_mandatory_source_overflow") + self.mandatory_source_overflow_visible = True + self.exercise_observations.add("mandatory_source_overflow_rejected_visible") + + def emit_frontend_scoped_event(self) -> None: + self.record("emit_frontend_scoped_event") + self.emit("proof_progress", scope="autonomous", phase=self.phase.value) + + def emit_context_overflow_contract_events(self) -> None: + self.record("emit_context_overflow_contract_events") + fatal_payload = { + "workflow_mode": "autonomous", + "role_id": "compiler_writer", + "configured_model": "configured-model", + "configured_provider": "openrouter", + "effective_model": "fallback-model", + "effective_provider": "lm_studio", + "fatal": True, + } + self.emit("context_overflow_error", **fatal_payload) + self.persisted_events.append( + WorkflowEvent(event_type="context_overflow_error", payload=dict(fatal_payload)) + ) + self.emit("auto_research_stopped", reason="context_overflow", **fatal_payload) + self.terminal_stop_events += 1 + self.emit( + "proof_context_overflow", + workflow_mode="autonomous", + scope="autonomous", + phase="brainstorm_proof_verification", + role_id="proof_formalization", + configured_model="configured-model", + configured_provider="openrouter", + effective_model="fallback-model", + effective_provider="lm_studio", + fatal=False, + ) + + def emit_registered_proof_verified(self, proof_id: str = "registered-proof-1") -> None: + self.record("emit_registered_proof_verified", proof_id=proof_id) + self.autonomous_proofs.add(proof_id) + self.emit("proof_verified", proof_id=proof_id, scope="autonomous", phase=self.phase.value) + + def force_paper_writing(self) -> None: + self.record("force_paper_writing") + if self.mode is WorkflowMode.AUTONOMOUS and self.phase is WorkflowPhase.TIER1_AGGREGATION: + self.parent_generation += 1 + self.stale_child_outputs_fenced = True + self.complete_brainstorm() + + def complete_paper(self) -> None: + self.record("complete_paper") + if self.mode is WorkflowMode.AUTONOMOUS and self.allow_research_papers: + if self.allow_mathematical_proofs and self.lean.enabled: + self.lean.run() + self.autonomous_proofs.add("paper-proof-1") + self.phase = WorkflowPhase.PAPER_PROOF + self.emit("proof_verified", proof_id="paper-proof-1", scope="autonomous", phase=self.phase.value) + self.phase = WorkflowPhase.TOPIC_EXPLORATION + self.checkpoint["phase"] = self.phase.value + self.emit("phase_changed", phase=self.phase.value) + + def run_manual_proof_check(self) -> None: + self.record("run_manual_proof_check") + if self.mode is not WorkflowMode.MANUAL_COMPILER: + return + self.phase = WorkflowPhase.MANUAL_PROOF + if self.lean.enabled: + self.lean.run() + self.manual_proofs_active.add("manual-proof-1") + self.emit("proof_verified", proof_id="manual-proof-1", scope="manual", phase=self.phase.value) + + def simulate_credit_exhaustion(self) -> None: + self.record("simulate_credit_exhaustion") + self.provider.exhaust_credit() + + def reset_credit(self) -> None: + self.record("reset_credit") + self.provider.reset_credit() + if self.phase is WorkflowPhase.PAUSED: + phase = self.checkpoint.get("resume_phase", WorkflowPhase.TOPIC_EXPLORATION.value) + self.phase = WorkflowPhase(phase) + self.checkpoint["paused"] = False + self.checkpoint["phase"] = self.phase.value + self.emit("provider_resumed", scope=self.mode.value, phase=self.phase.value) + + def toggle_session_history_memory(self, enabled: bool) -> None: + self.record("toggle_session_history_memory", enabled=enabled) + self.assistant.set_enabled(enabled) + + def refresh_assistant_pack(self, *proof_ids: str) -> None: + self.record("refresh_assistant_pack", proof_ids=proof_ids) + self.assistant.refresh_pack(*proof_ids) + + def validator_receive_assistant_pack(self) -> None: + self.record("validator_receive_assistant_pack") + self.assistant.inject_into_validator() + + def assistant_retrieve_non_blocking(self) -> None: + self.record("assistant_retrieve_non_blocking") + self.assistant.retrieve_non_blocking("prior-proof-1") + + def assistant_stagnant_pack_backoff(self) -> None: + self.record("assistant_stagnant_pack_backoff") + self.assistant.mark_stagnant_backoff() + + def stop(self) -> None: + self.record("stop") + if self.mode is WorkflowMode.NONE: + return + if self.phase is WorkflowPhase.PAUSED: + self.checkpoint.update({"mode": self.mode.value, "stopped": True}) + else: + self.checkpoint.update({"mode": self.mode.value, "phase": self.phase.value, "stopped": True}) + stopped_mode = self.mode + self.active_owners.discard(stopped_mode) + self.background_owners.discard(stopped_mode) + self.mode = WorkflowMode.NONE + self.emit("stopped") + + def resume(self) -> None: + self.record("resume") + if self.mode is not WorkflowMode.NONE or not self.checkpoint: + return + mode = self.checkpoint.get("mode") + phase = self.checkpoint.get("phase") + if mode: + self.mode = WorkflowMode(mode) + self.active_owners.add(self.mode) + if phase: + self.phase = WorkflowPhase(phase) + if self.mode is WorkflowMode.LEANOJ and self.leanoj_master_proof: + self.leanoj_draft_preserved_on_resume = True + self.checkpoint["stopped"] = False + self.emit("resumed", mode=self.mode.value, phase=self.phase.value) + + def clear(self) -> None: + self.record("clear") + self.manual_proofs_before_clear = set(self.manual_proofs_active) + self.manual_proofs_archived.update(self.manual_proofs_active) + self.manual_proofs_active.clear() + self.manual_aggregator_history.extend(self.manual_aggregator_submissions) + self.manual_aggregator_submissions.clear() + if self.mode is WorkflowMode.LEANOJ: + self.leanoj_cleared = True + self.leanoj_master_proof = "" + self.assistant.live_pack = () + self.lean.invocations = 0 + self.lean.blocked_invocations = 0 + self.smt.invocations = 0 + self.smt.blocked_invocations = 0 + self.active_state_cleared = True + self.history_preserved = True + self.pending_child_tasks = 0 + self.active_owners.clear() + self.background_owners.clear() + self.mode = WorkflowMode.NONE + self.phase = WorkflowPhase.IDLE + self.checkpoint = {} + self.emit("cleared") + + def resolve_runtime_path(self, relative_path: str = "state/checkpoint.json") -> None: + self.record("resolve_runtime_path", relative_path=relative_path) + resolved = (self.runtime_root / relative_path).resolve() + self.resolved_runtime_paths.append(resolved) + try: + resolved.relative_to(self.runtime_root.resolve()) + except ValueError: + self.runtime_root_violations += 1 + diff --git a/tests/workflow_harness/real_adapters/__init__.py b/tests/workflow_harness/real_adapters/__init__.py new file mode 100644 index 0000000..63f289b --- /dev/null +++ b/tests/workflow_harness/real_adapters/__init__.py @@ -0,0 +1,72 @@ +"""Test-only adapters for driving real MOTO workflow boundaries. + +Exports are lazy so metadata-only registry/deep-run collection does not import production +coordinators. Executing a real adapter still imports the same production boundary normally. +""" + +from importlib import import_module + +_EXPORT_MODULES = { + "EventCollector": "dependency_fakes", + "FakeProofStage": "dependency_fakes", + "FakeResearchMetadata": "dependency_fakes", + "RoleConfigCapture": "dependency_fakes", + "route_workflow_state": "dependency_fakes", + "RealWorkflowObservation": "observed_state", + "ManualAggregatorAdapter": "manual_aggregator_adapter", + "minimal_aggregator_request": "manual_aggregator_adapter", + "ManualCompilerAdapter": "manual_compiler_adapter", + "minimal_compiler_request": "manual_compiler_adapter", + "LeanOJAdapter": "leanoj_adapter", + "minimal_leanoj_request": "leanoj_adapter", + "StartOutcome": "cross_mode_adapter", + "assert_single_race_winner": "cross_mode_adapter", + "race_starts": "cross_mode_adapter", + "FatalContextOverflowEvent": "event_assertions", + "ProofContextOverflowEvent": "event_assertions", + "assert_event_count": "event_assertions", + "assert_fatal_context_overflow_event": "event_assertions", + "assert_no_events": "event_assertions", + "assert_proof_context_overflow_event": "event_assertions", + "assert_route_identity": "event_assertions", + "event_payloads": "event_assertions", + "SourceTaggedDocument": "source_tagged_rag", + "SourceTaggedRagIndex": "source_tagged_rag", +} + + +def __getattr__(name: str): + module_name = _EXPORT_MODULES.get(name) + if module_name is None: + raise AttributeError(name) + value = getattr(import_module(f"{__name__}.{module_name}"), name) + globals()[name] = value + return value + +__all__ = [ + "EventCollector", + "FakeProofStage", + "FakeResearchMetadata", + "RealWorkflowObservation", + "RoleConfigCapture", + "route_workflow_state", + "ManualAggregatorAdapter", + "minimal_aggregator_request", + "ManualCompilerAdapter", + "minimal_compiler_request", + "LeanOJAdapter", + "minimal_leanoj_request", + "StartOutcome", + "assert_single_race_winner", + "race_starts", + "FatalContextOverflowEvent", + "ProofContextOverflowEvent", + "assert_event_count", + "assert_fatal_context_overflow_event", + "assert_no_events", + "assert_proof_context_overflow_event", + "assert_route_identity", + "event_payloads", + "SourceTaggedDocument", + "SourceTaggedRagIndex", +] diff --git a/tests/workflow_harness/real_adapters/cross_mode_adapter.py b/tests/workflow_harness/real_adapters/cross_mode_adapter.py new file mode 100644 index 0000000..558d747 --- /dev/null +++ b/tests/workflow_harness/real_adapters/cross_mode_adapter.py @@ -0,0 +1,48 @@ +"""Test-only helpers for serialized cross-mode start races.""" +from __future__ import annotations + +import asyncio +from dataclasses import dataclass +from typing import Awaitable, Callable + +from fastapi import HTTPException + + +StartCall = Callable[[], Awaitable[dict]] + + +@dataclass(frozen=True) +class StartOutcome: + name: str + started: bool + status_code: int | None = None + detail: str = "" + + +async def race_starts(calls: dict[str, StartCall]) -> tuple[StartOutcome, ...]: + """Run route starts concurrently and normalize their observable outcomes.""" + + async def run(name: str, call: StartCall) -> StartOutcome: + try: + await call() + return StartOutcome(name=name, started=True) + except HTTPException as exc: + return StartOutcome( + name=name, + started=False, + status_code=exc.status_code, + detail=str(exc.detail), + ) + + outcomes = await asyncio.gather(*(run(name, call) for name, call in calls.items())) + return tuple(outcomes) + + +def assert_single_race_winner(outcomes: tuple[StartOutcome, ...]) -> StartOutcome: + winners = [outcome for outcome in outcomes if outcome.started] + if len(winners) != 1: + raise AssertionError(f"Expected one start winner, observed {outcomes!r}") + losers = [outcome for outcome in outcomes if not outcome.started] + if any(outcome.status_code != 400 for outcome in losers): + raise AssertionError(f"Expected conflict HTTP 400 for all losers, observed {outcomes!r}") + return winners[0] diff --git a/tests/workflow_harness/real_adapters/deep_bootstrap.py b/tests/workflow_harness/real_adapters/deep_bootstrap.py new file mode 100644 index 0000000..72de419 --- /dev/null +++ b/tests/workflow_harness/real_adapters/deep_bootstrap.py @@ -0,0 +1,46 @@ +"""Early pytest bootstrap assertions for contained deep real-adapter children.""" +from __future__ import annotations + +import os +from pathlib import Path + + +def _within(path: str | Path, root: Path) -> bool: + candidate = Path(path).resolve() + try: + candidate.relative_to(root.resolve()) + return True + except ValueError: + return False + + +def pytest_sessionstart(session) -> None: # pragma: no cover - executed in child pytest + del session + from backend.shared.config import system_config + + expected_data = Path(os.environ["MOTO_DATA_ROOT"]).resolve() + expected_logs = Path(os.environ["MOTO_LOG_ROOT"]).resolve() + assert Path(system_config.data_dir).resolve() == expected_data + assert Path(system_config.logs_dir or "").resolve() == expected_logs + + data_paths = ( + system_config.user_uploads_dir, + system_config.chroma_db_dir, + system_config.shared_training_file, + system_config.compiler_outline_file, + system_config.compiler_paper_file, + system_config.compiler_rejections_file, + system_config.compiler_acceptances_file, + system_config.compiler_declines_file, + system_config.auto_brainstorms_dir, + system_config.auto_papers_dir, + system_config.auto_papers_archive_dir, + system_config.auto_research_metadata_file, + system_config.auto_research_stats_file, + system_config.auto_workflow_state_file, + system_config.auto_research_topic_rejections_file, + system_config.auto_sessions_base_dir, + system_config.lean4_workspace_dir, + ) + escaped = [str(path) for path in data_paths if path and not _within(path, expected_data)] + assert not escaped, f"Deep child has mutable paths outside MOTO_DATA_ROOT: {escaped}" diff --git a/tests/workflow_harness/real_adapters/deep_runner.py b/tests/workflow_harness/real_adapters/deep_runner.py new file mode 100644 index 0000000..7571c4a --- /dev/null +++ b/tests/workflow_harness/real_adapters/deep_runner.py @@ -0,0 +1,261 @@ +"""Contained subprocess runner used by the opt-in Build F deep matrix.""" +from __future__ import annotations + +from dataclasses import dataclass +import hashlib +import os +from pathlib import Path +import re +import shutil +import signal +import subprocess +import sys +from typing import Iterable + +from .maturity_registry import RealAdapterDescriptor, RepeatContract + + +_PASSTHROUGH_ENV = ( + "COMSPEC", + "LANG", + "LC_ALL", + "PATH", + "PATHEXT", + "SYSTEMDRIVE", + "SYSTEMROOT", + "WINDIR", +) +_WORKSPACE_MUTABLE_ROOTS = ( + "backend/data", + "backend/logs", + ".moto_instances", + ".moto_launcher_state.json", + ".moto_last_instance.json", +) +_IGNORED_NAMES = {"__pycache__", ".pytest_cache"} + + +@dataclass(frozen=True) +class DeepRunObservation: + variant: str + hash_seed: str + returncode: int + normalized_output: str + runtime_state_digest: str + stdout: str + stderr: str + workspace_changes: tuple[str, ...] = () + + def repeat_value(self, contract: RepeatContract) -> tuple[object, ...]: + base = (self.returncode, self.normalized_output) + if contract is RepeatContract.NORMALIZED_PROCESS_AND_RUNTIME_STATE: + return (*base, self.runtime_state_digest) + return base + + +def minimal_deep_environment( + base_environment: dict[str, str], + *, + workspace_root: Path, + variant_root: Path, + fake_root: Path, + hash_seed: str, +) -> dict[str, str]: + """Build an allowlisted environment with provider/proxy/keyring access disabled.""" + env = { + key: base_environment[key] + for key in _PASSTHROUGH_ENV + if base_environment.get(key) + } + home = variant_root / "home" + temp_root = variant_root / "tmp" + env.update( + { + "HOME": str(home), + "USERPROFILE": str(home), + "TEMP": str(temp_root), + "TMP": str(temp_root), + "MOTO_REAL_ADAPTER_DEEP_TESTS": "0", + "MOTO_WORKFLOW_DEEP_TESTS": "0", + "MOTO_DATA_ROOT": str(variant_root / "data"), + "MOTO_LOG_ROOT": str(variant_root / "logs"), + "MOTO_GENERIC_MODE": "false", + "MOTO_LM_STUDIO_BASE_URL": "http://127.0.0.1:9", + "MOTO_LEAN4_ENABLED": "false", + "MOTO_SMT_ENABLED": "false", + "MOTO_SECRET_NAMESPACE": f"build-f-{hash_seed}", + "PYTHONHASHSEED": hash_seed, + "PYTHONDONTWRITEBYTECODE": "1", + "PYTHON_KEYRING_BACKEND": "keyring.backends.null.Keyring", + "PLAYWRIGHT_BROWSERS_PATH": str(variant_root / "no-browser-runtime"), + "NO_PROXY": "127.0.0.1,localhost,::1", + "no_proxy": "127.0.0.1,localhost,::1", + "PYTHONPATH": os.pathsep.join((str(fake_root), str(workspace_root))), + } + ) + return env + + +def snapshot_paths(paths: Iterable[Path], *, max_depth: int | None = None) -> dict[str, str]: + snapshot: dict[str, str] = {} + for root in paths: + if not root.exists(): + snapshot[str(root)] = "<absent>" + continue + candidates = (root,) if root.is_file() else root.rglob("*") + for path in candidates: + if max_depth is not None: + try: + depth = len(path.relative_to(root).parts) + except ValueError: + continue + if depth > max_depth: + continue + if any(part in _IGNORED_NAMES for part in path.parts) or not path.is_file(): + continue + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + snapshot[str(path)] = digest.hexdigest() + return snapshot + + +def workspace_mutable_snapshot(workspace_root: Path) -> dict[str, str]: + # Runtime roots can contain very large historical corpora. Two levels catches accidental + # fallback writes and root-state mutations without turning every deep case into a full data + # integrity scan; individual scenario adapters still assert their concrete artifacts' roots. + return snapshot_paths( + (workspace_root / relative for relative in _WORKSPACE_MUTABLE_ROOTS), + max_depth=2, + ) + + +def runtime_state_digest(variant_root: Path) -> str: + state = snapshot_paths((variant_root / "data", variant_root / "logs")) + normalized = "\n".join( + f"{Path(path).relative_to(variant_root)}={digest}" + for path, digest in sorted(state.items()) + ) + return hashlib.sha256(normalized.encode("utf-8")).hexdigest() + + +def snapshot_diff(before: dict[str, str], after: dict[str, str]) -> tuple[str, ...]: + """Return a stable, content-based description of mutable workspace changes.""" + changes: list[str] = [] + for path in sorted(set(before) | set(after)): + old = before.get(path) + new = after.get(path) + if old == new: + continue + if old is None: + kind = "added" + elif new is None: + kind = "deleted" + else: + kind = "modified" + changes.append(f"{kind}: {path}") + return tuple(changes) + + +def normalize_pytest_output(text: str, roots: Iterable[Path]) -> str: + normalized = text.replace("\\", "/") + for root in roots: + normalized = normalized.replace(str(root).replace("\\", "/"), "<ROOT>") + normalized = re.sub(r"\b\d+(?:\.\d+)?s\b", "<TIME>", normalized) + normalized = re.sub(r"\bin \d+(?:\.\d+)? seconds?\b", "in <TIME>", normalized) + return "\n".join(line.rstrip() for line in normalized.splitlines() if line.strip()) + + +def _terminate_process_tree(process: subprocess.Popen[str]) -> None: + if process.poll() is not None: + return + if os.name == "nt": + subprocess.run( + ["taskkill", "/PID", str(process.pid), "/T", "/F"], + capture_output=True, + check=False, + timeout=15, + ) + else: + try: + os.killpg(process.pid, signal.SIGKILL) + except ProcessLookupError: + pass + try: + process.wait(timeout=10) + except subprocess.TimeoutExpired: + process.kill() + + +def run_deep_variant( + descriptor: RealAdapterDescriptor, + *, + workspace_root: Path, + variant_root: Path, + fake_root: Path, + variant: str, + hash_seed: str, +) -> DeepRunObservation: + workspace_before = workspace_mutable_snapshot(workspace_root) + env = minimal_deep_environment( + os.environ, + workspace_root=workspace_root, + variant_root=variant_root, + fake_root=fake_root, + hash_seed=hash_seed, + ) + command = [ + sys.executable, + "-m", + "pytest", + *descriptor.pytest_args(), + "-p", + "no:cacheprovider", + "-p", + "tests.workflow_harness.real_adapters.deep_bootstrap", + "--basetemp", + str(variant_root / "pytest-temp"), + ] + popen_kwargs: dict[str, object] = {} + if os.name == "nt": + popen_kwargs["creationflags"] = subprocess.CREATE_NEW_PROCESS_GROUP + else: + popen_kwargs["start_new_session"] = True + process = subprocess.Popen( + command, + cwd=workspace_root, + env=env, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + **popen_kwargs, + ) + try: + stdout, stderr = process.communicate(timeout=descriptor.timeout_seconds) + except subprocess.TimeoutExpired as exc: + _terminate_process_tree(process) + stdout, stderr = process.communicate() + raise AssertionError( + f"{descriptor.scenario_id} ({variant}) exceeded " + f"{descriptor.timeout_seconds}s\nSTDOUT:\n{stdout}\nSTDERR:\n{stderr}" + ) from exc + combined = f"{stdout}\n{stderr}" + workspace_after = workspace_mutable_snapshot(workspace_root) + return DeepRunObservation( + variant=variant, + hash_seed=hash_seed, + returncode=process.returncode, + normalized_output=normalize_pytest_output( + combined, (workspace_root, variant_root) + ), + runtime_state_digest=runtime_state_digest(variant_root), + stdout=stdout, + stderr=stderr, + workspace_changes=snapshot_diff(workspace_before, workspace_after), + ) + + +def remove_variant_root(variant_root: Path) -> None: + if variant_root.exists(): + shutil.rmtree(variant_root) diff --git a/tests/workflow_harness/real_adapters/dependency_fakes.py b/tests/workflow_harness/real_adapters/dependency_fakes.py new file mode 100644 index 0000000..cfb5605 --- /dev/null +++ b/tests/workflow_harness/real_adapters/dependency_fakes.py @@ -0,0 +1,129 @@ +from __future__ import annotations + +from dataclasses import dataclass, field +from types import SimpleNamespace +from typing import Any + +from backend.autonomous.core.proof_verification_stage import ProofVerificationProviderPause +from backend.shared.models import ProofCandidate, ProofStageResult + + +@dataclass +class EventCollector: + events: list[tuple[str, dict[str, Any]]] = field(default_factory=list) + + async def broadcast(self, event_type: str, payload: dict[str, Any] | None = None) -> None: + self.events.append((event_type, dict(payload or {}))) + + def payloads(self, event_type: str) -> list[dict[str, Any]]: + return [payload for current_type, payload in self.events if current_type == event_type] + + +@dataclass +class RoleConfigCapture: + roles: dict[str, Any] = field(default_factory=dict) + + def configure(self, role_id: str, config: Any) -> None: + self.roles[role_id] = config + + +class FakeProofStage: + def __init__( + self, + *, + result: ProofStageResult | None = None, + pause: bool = False, + checkpoint: dict[str, Any] | None = None, + ) -> None: + self.result = result + self.pause = pause + self.checkpoint = checkpoint + self.calls: list[dict[str, Any]] = [] + self.pause_count = 0 + + async def run(self, **kwargs: Any) -> ProofStageResult: + self.calls.append(kwargs) + if self.checkpoint and kwargs.get("checkpoint_callback"): + await kwargs["checkpoint_callback"](dict(self.checkpoint)) + if self.pause and self.pause_count == 0: + self.pause_count += 1 + raise ProofVerificationProviderPause( + "OpenRouter credit exhausted", + remaining_candidates=[ + ProofCandidate( + theorem_id="remaining_candidate", + statement="A remaining prompt-relevant theorem.", + expected_novelty_tier="novel_variant", + ) + ], + ) + if kwargs.get("append_proof_callback"): + await kwargs["append_proof_callback"]( + { + "proof_id": "manual-proof-1", + "source_type": kwargs.get("source_type"), + "source_id": kwargs.get("source_id"), + } + ) + return self.result or ProofStageResult( + source_type=kwargs["source_type"], + source_id=kwargs["source_id"], + total_candidates=1, + verified_count=1, + novel_count=1, + ) + + +class FakeResearchMetadata: + def __init__(self) -> None: + self.proof_checkpoint: dict[str, Any] | None = None + self.saved_proof_checkpoints: list[dict[str, Any]] = [] + self.workflow_states: list[dict[str, Any]] = [] + self.completed_triggers: list[str] = [] + + async def save_proof_checkpoint(self, checkpoint: dict[str, Any]) -> None: + self.proof_checkpoint = dict(checkpoint) + self.saved_proof_checkpoints.append(dict(checkpoint)) + + async def get_proof_checkpoint( + self, + source_type: str | None = None, + source_id: str | None = None, + trigger: str | None = None, + ) -> dict[str, Any] | None: + if not self.proof_checkpoint: + return None + if source_type and self.proof_checkpoint.get("source_type") != source_type: + return None + if source_id and self.proof_checkpoint.get("source_id") != source_id: + return None + if trigger and self.proof_checkpoint.get("trigger") != trigger: + return None + return dict(self.proof_checkpoint) + + async def mark_proof_checkpoint_trigger_complete( + self, + source_type: str, + source_id: str, + trigger: str, + source_title: str = "", + ) -> None: + self.completed_triggers.append(trigger) + self.proof_checkpoint = { + **(self.proof_checkpoint or {}), + "source_type": source_type, + "source_id": source_id, + "source_title": source_title, + "trigger": trigger, + "status": "trigger_complete", + } + + async def save_workflow_state(self, state: dict[str, Any]) -> None: + self.workflow_states.append(dict(state)) + + async def set_current_brainstorm(self, _topic_id: str | None) -> None: + return None + + +def route_workflow_state(*, is_running: bool = False, current_tier: str = "idle") -> SimpleNamespace: + return SimpleNamespace(is_running=is_running, current_tier=current_tier, is_active=is_running) diff --git a/tests/workflow_harness/real_adapters/event_assertions.py b/tests/workflow_harness/real_adapters/event_assertions.py new file mode 100644 index 0000000..dbd975a --- /dev/null +++ b/tests/workflow_harness/real_adapters/event_assertions.py @@ -0,0 +1,121 @@ +from __future__ import annotations + +from collections.abc import Mapping, Sequence +from typing import Any, TypedDict + + +class FatalContextOverflowEvent(TypedDict, total=False): + workflow_mode: str + role_id: str + configured_model: str + configured_provider: str + effective_model: str + effective_provider: str + effective_host_provider: str + route_kind: str + reason: str + message: str + resolution: str + fatal: bool + + +class ProofContextOverflowEvent(FatalContextOverflowEvent, total=False): + source_type: str + source_id: str + theorem_id: str + overflow_origin: str + prompt_tokens: int + max_input_tokens: int + + +EventRecord = tuple[str, Mapping[str, Any]] + + +def event_payloads( + events: Sequence[EventRecord], + event_type: str, +) -> list[Mapping[str, Any]]: + return [payload for current_type, payload in events if current_type == event_type] + + +def assert_event_count( + events: Sequence[EventRecord], + event_type: str, + expected: int, +) -> list[Mapping[str, Any]]: + payloads = event_payloads(events, event_type) + assert len(payloads) == expected, ( + f"expected {expected} {event_type!r} event(s), found {len(payloads)}: {payloads!r}" + ) + return payloads + + +def assert_no_events(events: Sequence[EventRecord], *event_types: str) -> None: + for event_type in event_types: + assert_event_count(events, event_type, 0) + + +def assert_fatal_context_overflow_event( + payload: Mapping[str, Any], + *, + workflow_mode: str, + role_id: str, +) -> FatalContextOverflowEvent: + required = { + "workflow_mode", + "role_id", + "configured_model", + "configured_provider", + "reason", + "message", + "resolution", + } + missing = required.difference(payload) + assert not missing, f"context_overflow_error missing fields: {sorted(missing)}" + assert payload["workflow_mode"] == workflow_mode + assert payload["role_id"] == role_id + assert payload["reason"] == "context_overflow" + assert payload.get("fatal", True) is True + return dict(payload) # type: ignore[return-value] + + +def assert_proof_context_overflow_event( + payload: Mapping[str, Any], + *, + workflow_mode: str, +) -> ProofContextOverflowEvent: + required = { + "workflow_mode", + "source_type", + "source_id", + "configured_model", + "configured_provider", + "overflow_origin", + "message", + "fatal", + } + missing = required.difference(payload) + assert not missing, f"proof_context_overflow missing fields: {sorted(missing)}" + assert payload["workflow_mode"] == workflow_mode + assert payload["fatal"] is False + return dict(payload) # type: ignore[return-value] + + +def assert_route_identity( + payload: Mapping[str, Any], + *, + configured_model: str, + configured_provider: str, + effective_model: str, + effective_provider: str, + effective_host_provider: str | None = None, + route_kind: str | None = None, +) -> None: + assert payload["configured_model"] == configured_model + assert payload["configured_provider"] == configured_provider + assert payload["effective_model"] == effective_model + assert payload["effective_provider"] == effective_provider + if effective_host_provider is not None: + assert payload["effective_host_provider"] == effective_host_provider + if route_kind is not None: + assert payload["route_kind"] == route_kind diff --git a/tests/workflow_harness/real_adapters/filesystem_assertions.py b/tests/workflow_harness/real_adapters/filesystem_assertions.py new file mode 100644 index 0000000..a104221 --- /dev/null +++ b/tests/workflow_harness/real_adapters/filesystem_assertions.py @@ -0,0 +1,30 @@ +"""Reusable filesystem assertions for real-adapter workflow tests.""" +from __future__ import annotations + +from pathlib import Path +from typing import Iterable + + +def assert_paths_within(root: Path, paths: Iterable[Path]) -> None: + """Assert that every supplied path is contained by the temporary runtime root.""" + resolved_root = root.resolve() + for path in paths: + assert path.resolve().is_relative_to(resolved_root), ( + f"{path} escaped temporary runtime root {resolved_root}" + ) + + +def assert_files_exist(root: Path, *relative_paths: str) -> None: + """Assert that the named files exist below the supplied runtime root.""" + paths = [root / relative_path for relative_path in relative_paths] + assert_paths_within(root, paths) + for path in paths: + assert path.is_file(), f"Expected workflow file does not exist: {path}" + + +def assert_paths_absent(root: Path, *relative_paths: str) -> None: + """Assert that the named files or directories are absent below the runtime root.""" + paths = [root / relative_path for relative_path in relative_paths] + assert_paths_within(root, paths) + for path in paths: + assert not path.exists(), f"Workflow path should have been removed: {path}" diff --git a/tests/workflow_harness/real_adapters/leanoj_adapter.py b/tests/workflow_harness/real_adapters/leanoj_adapter.py new file mode 100644 index 0000000..bb7324d --- /dev/null +++ b/tests/workflow_harness/real_adapters/leanoj_adapter.py @@ -0,0 +1,57 @@ +"""Thin test-only driver for real LeanOJ routes and coordinator state.""" +from __future__ import annotations + +from dataclasses import dataclass +from typing import Any + +from backend.api.routes import leanoj as leanoj_route +from backend.leanoj.core.leanoj_coordinator import LeanOJCoordinator +from backend.shared.models import LeanOJRoleConfig, LeanOJStartRequest + + +def minimal_leanoj_request(**overrides: Any) -> LeanOJStartRequest: + role = LeanOJRoleConfig( + model_id="leanoj-test-model", + context_window=8192, + max_output_tokens=1024, + ) + values: dict[str, Any] = { + "user_prompt": "Prove one equals one.", + "lean_template": "import Mathlib\n\nexample : 1 = 1 := by\n sorry", + "topic_generator": role, + "topic_validator": role, + "brainstorm_submitters": [role], + "brainstorm_validator": role, + "path_decider": role, + "final_solver": role, + } + values.update(overrides) + return LeanOJStartRequest(**values) + + +@dataclass +class LeanOJAdapter: + """Route actions plus bounded direct coordinator operations.""" + + route: Any = leanoj_route + coordinator: LeanOJCoordinator | None = None + + async def start(self, request: LeanOJStartRequest | None = None) -> dict[str, Any]: + return await self.route.start_leanoj(request or minimal_leanoj_request()) + + async def stop(self) -> dict[str, Any]: + return await self.route.stop_leanoj() + + async def clear(self, *, confirm: bool = True) -> dict[str, Any]: + return await self.route.clear_leanoj(confirm=confirm) + + async def skip_brainstorm(self) -> dict[str, Any]: + return await self.route.skip_leanoj_brainstorm() + + async def force_brainstorm(self) -> dict[str, Any]: + return await self.route.force_leanoj_brainstorm() + + async def write_intermediate_master_proof(self, content: str, *, summary: str) -> None: + if self.coordinator is None: + raise RuntimeError("A coordinator is required for direct master-proof edits.") + await self.coordinator._write_master_proof(content, summary=summary) diff --git a/tests/workflow_harness/real_adapters/manual_aggregator_adapter.py b/tests/workflow_harness/real_adapters/manual_aggregator_adapter.py new file mode 100644 index 0000000..35f42af --- /dev/null +++ b/tests/workflow_harness/real_adapters/manual_aggregator_adapter.py @@ -0,0 +1,48 @@ +"""Thin test-only driver for the real manual Aggregator route lifecycle.""" +from __future__ import annotations + +from dataclasses import dataclass +from typing import Any + +from backend.api.routes import aggregator as aggregator_route +from backend.shared.models import AggregatorStartRequest, SubmitterConfig + + +def minimal_aggregator_request(**overrides: Any) -> AggregatorStartRequest: + values: dict[str, Any] = { + "user_prompt": "Build a rigorous solution.", + "submitter_configs": [ + SubmitterConfig( + submitter_id=1, + provider="lm_studio", + model_id="test-submitter", + context_window=4096, + max_output_tokens=512, + ) + ], + "validator_provider": "lm_studio", + "validator_model": "test-validator", + "validator_context_size": 4096, + "validator_max_output_tokens": 512, + } + values.update(overrides) + return AggregatorStartRequest(**values) + + +@dataclass +class ManualAggregatorAdapter: + """Calls production route functions while dependencies are patched by tests.""" + + route: Any = aggregator_route + + async def start(self, request: AggregatorStartRequest | None = None) -> dict[str, Any]: + return await self.route.start_aggregator(request or minimal_aggregator_request()) + + async def stop(self) -> dict[str, Any]: + return await self.route.stop_aggregator() + + async def clear(self) -> dict[str, Any]: + return await self.route.clear_all_submissions() + + async def save_results(self) -> dict[str, Any]: + return await self.route.save_results() diff --git a/tests/workflow_harness/real_adapters/manual_compiler_adapter.py b/tests/workflow_harness/real_adapters/manual_compiler_adapter.py new file mode 100644 index 0000000..c4b0505 --- /dev/null +++ b/tests/workflow_harness/real_adapters/manual_compiler_adapter.py @@ -0,0 +1,44 @@ +"""Thin test-only driver for the real manual Compiler route lifecycle.""" +from __future__ import annotations + +from dataclasses import dataclass +from typing import Any + +from backend.api.routes import compiler as compiler_route +from backend.shared.models import CompilerStartRequest + + +def minimal_compiler_request(**overrides: Any) -> CompilerStartRequest: + values: dict[str, Any] = { + "compiler_prompt": "Write a rigorous paper.", + "validator_model": "test-validator", + "validator_context_size": 4096, + "validator_max_output_tokens": 512, + "writer_model": "test-writer", + "writer_context_size": 4096, + "writer_max_output_tokens": 512, + "high_param_model": "test-rigor", + "high_param_context_size": 4096, + "high_param_max_output_tokens": 512, + } + values.update(overrides) + return CompilerStartRequest(**values) + + +@dataclass +class ManualCompilerAdapter: + """Calls production route functions while dependencies are patched by tests.""" + + route: Any = compiler_route + + async def start(self, request: CompilerStartRequest | None = None) -> dict[str, Any]: + return await self.route.start_compiler(request or minimal_compiler_request()) + + async def stop(self) -> dict[str, Any]: + return await self.route.stop_compiler() + + async def clear(self, *, confirm: bool = True) -> dict[str, Any]: + return await self.route.clear_paper(confirm=confirm) + + async def save(self) -> dict[str, Any]: + return await self.route.save_paper() diff --git a/tests/workflow_harness/real_adapters/maturity_registry.py b/tests/workflow_harness/real_adapters/maturity_registry.py new file mode 100644 index 0000000..1c2cacb --- /dev/null +++ b/tests/workflow_harness/real_adapters/maturity_registry.py @@ -0,0 +1,222 @@ +"""Build F maturity registry for the test-only real-adapter overlay.""" +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum +from pathlib import Path +from typing import Iterable + + +class AdapterMaturity(str, Enum): + NORMAL = "normal" + DEEP = "deep" + BLOCKED = "blocked" + + +class RepeatContract(str, Enum): + """What a repeated deep run can truthfully compare.""" + + NORMALIZED_PROCESS_RESULT = "normalized_process_result" + NORMALIZED_PROCESS_AND_RUNTIME_STATE = "normalized_process_and_runtime_state" + + +@dataclass(frozen=True) +class CleanupContract: + uses_temporary_roots: bool + removes_runtime_state: bool + forbids_workspace_escape: bool + notes: str + + +@dataclass(frozen=True) +class RealAdapterDescriptor: + scenario_id: str + adapter: str + maturity: AdapterMaturity + timeout_seconds: int + repeat_safe: bool + fields: tuple[str, ...] + invariants: tuple[str, ...] + cleanup: CleanupContract + test_selectors: tuple[str, ...] + repeat_contract: RepeatContract | None + repeat_contract_reason: str | None + blocked_reason: str | None = None + + @property + def executable(self) -> bool: + return bool(self.test_selectors) + + def pytest_args(self) -> tuple[str, ...]: + if not self.test_selectors: + raise ValueError(f"{self.scenario_id} is metadata-only") + return ("-q", *self.test_selectors) + + +_DEEP_IDS = { + "real_actual_route_start_conflict_matrix_no_side_effects", + "real_shared_start_guard_representative_race", + "real_proof_scope_matrix_isolated_under_temp_root", + "real_direct_source_rag_exclusion_matrix", + "real_model_visible_prompts_strip_generated_proof_appendices", + "real_rigor_mandatory_source_overflow_fails_before_model_or_rag", +} + +_RUNTIME_STATE_REPEAT_IDS = { + "real_model_visible_prompts_strip_generated_proof_appendices", + "real_rigor_mandatory_source_overflow_fails_before_model_or_rag", +} + +_BLOCKED_IDS = { + "real_parent_action_fencing_unavailable_without_production_seam", + "real_provider_stop_reset_checkpoint_unavailable_without_wait_seam", + "real_leanoj_full_final_loop_not_safely_bounded", +} + +_DEEP_CLEANUP = CleanupContract( + uses_temporary_roots=True, + removes_runtime_state=True, + forbids_workspace_escape=True, + notes=( + "Each repeated subprocess receives isolated data/log/temp roots; the matrix " + "checks guarded workspace paths before and after execution and removes the variant root." + ), +) +_NORMAL_CLEANUP = CleanupContract( + uses_temporary_roots=True, + removes_runtime_state=True, + forbids_workspace_escape=True, + notes="The selected Build B-E test owns cleanup through pytest tmp_path and adapter fixtures.", +) +_BLOCKED_CLEANUP = CleanupContract( + uses_temporary_roots=False, + removes_runtime_state=False, + forbids_workspace_escape=True, + notes="Metadata-only: no production transition owner or external dependency is invoked.", +) + + +def _build_registry() -> tuple[RealAdapterDescriptor, ...]: + # Imported lazily to keep the removable adapter package independent of coverage reporting. + from tests.workflow_real_adapters.coverage_records import REAL_ADAPTER_COVERAGE + + descriptors: list[RealAdapterDescriptor] = [] + for record in REAL_ADAPTER_COVERAGE: + blocked = record.result == "blocked" + deep = record.scenario_id in _DEEP_IDS + maturity = ( + AdapterMaturity.BLOCKED + if blocked + else AdapterMaturity.DEEP + if deep + else AdapterMaturity.NORMAL + ) + diagnostics = record.diagnostics or {} + descriptors.append( + RealAdapterDescriptor( + scenario_id=record.scenario_id, + adapter=record.adapter, + maturity=maturity, + timeout_seconds=0 if blocked else 180 if deep else 60, + repeat_safe=not blocked, + fields=tuple(record.fields), + invariants=tuple(record.invariants), + cleanup=( + _BLOCKED_CLEANUP + if blocked + else _DEEP_CLEANUP + if deep + else _NORMAL_CLEANUP + ), + test_selectors=tuple(record.test_selectors), + repeat_contract=( + ( + RepeatContract.NORMALIZED_PROCESS_AND_RUNTIME_STATE + if record.scenario_id in _RUNTIME_STATE_REPEAT_IDS + else RepeatContract.NORMALIZED_PROCESS_RESULT + ) + if deep + else None + ), + repeat_contract_reason=( + ( + "The scenario promises deterministic logical runtime artifacts." + if record.scenario_id in _RUNTIME_STATE_REPEAT_IDS + else "The scenario creates timestamped or run-identified state; repeatability is the normalized process result." + ) + if deep + else None + ), + blocked_reason=str(diagnostics.get("reason", "")) or None, + ) + ) + return tuple(descriptors) + + +REAL_ADAPTER_MATURITY_REGISTRY = _build_registry() + + +def descriptors_for( + maturity: AdapterMaturity, +) -> tuple[RealAdapterDescriptor, ...]: + return tuple( + descriptor + for descriptor in REAL_ADAPTER_MATURITY_REGISTRY + if descriptor.maturity is maturity + ) + + +def validate_maturity_registry( + descriptors: Iterable[RealAdapterDescriptor] = REAL_ADAPTER_MATURITY_REGISTRY, + *, + workspace_root: Path | None = None, +) -> None: + records = tuple(descriptors) + ids = [record.scenario_id for record in records] + assert len(ids) == len(set(ids)), "real-adapter scenario IDs must be unique" + from tests.workflow_real_adapters.coverage_records import REAL_ADAPTER_COVERAGE + + coverage_by_id = {record.scenario_id: record for record in REAL_ADAPTER_COVERAGE} + assert set(ids) == set(coverage_by_id), "registry must exactly cover coverage results" + assert _DEEP_IDS.isdisjoint(_BLOCKED_IDS) + assert set(_DEEP_IDS) == { + record.scenario_id + for record in records + if record.maturity is AdapterMaturity.DEEP + } + assert set(_BLOCKED_IDS) == { + record.scenario_id + for record in records + if record.maturity is AdapterMaturity.BLOCKED + } + for record in records: + coverage = coverage_by_id[record.scenario_id] + assert (record.maturity is AdapterMaturity.BLOCKED) == ( + coverage.result == "blocked" + ) + assert coverage.result in {"passed", "blocked"} + assert record.adapter in {"real_route", "real_coordinator"} + assert record.fields and record.invariants + assert record.timeout_seconds >= 0 + assert record.cleanup.forbids_workspace_escape + if record.maturity is AdapterMaturity.BLOCKED: + assert not record.executable + assert not record.repeat_safe + assert record.blocked_reason + assert record.repeat_contract is None + assert record.repeat_contract_reason is None + continue + assert record.executable and record.repeat_safe + assert record.timeout_seconds > 0 + if record.maturity is AdapterMaturity.DEEP: + assert record.repeat_contract is not None + assert record.repeat_contract_reason + assert record.test_selectors + assert len(record.test_selectors) == len(set(record.test_selectors)) + for selector in record.test_selectors: + assert selector.count("::") == 1, selector + assert selector.split("::", 1)[1].startswith("test_"), selector + if workspace_root is not None: + for selector in record.test_selectors: + selector_path = selector.split("::", 1)[0] + assert (workspace_root / selector_path).is_file(), selector_path diff --git a/tests/workflow_harness/real_adapters/observed_state.py b/tests/workflow_harness/real_adapters/observed_state.py new file mode 100644 index 0000000..7984736 --- /dev/null +++ b/tests/workflow_harness/real_adapters/observed_state.py @@ -0,0 +1,117 @@ +from __future__ import annotations + +from dataclasses import dataclass, field +from pathlib import Path + +from tests.workflow_harness.model import ( + FakeAssistantMemory, + FakeLean, + FakeProvider, + FakeSmt, + WorkflowEvent, + WorkflowMode, + WorkflowPhase, +) + + +@dataclass +class RealWorkflowObservation: + """Small invariant-compatible view over real route/coordinator behavior.""" + + runtime_root: Path + mode: WorkflowMode = WorkflowMode.NONE + phase: WorkflowPhase = WorkflowPhase.IDLE + previous_phase: WorkflowPhase = WorkflowPhase.IDLE + allow_mathematical_proofs: bool = True + allow_research_papers: bool = True + provider: FakeProvider = field(default_factory=FakeProvider) + lean: FakeLean = field(default_factory=FakeLean) + smt: FakeSmt = field(default_factory=FakeSmt) + assistant: FakeAssistantMemory = field(default_factory=FakeAssistantMemory) + autonomous_proofs: set[str] = field(default_factory=set) + manual_proofs_active: set[str] = field(default_factory=set) + manual_proofs_archived: set[str] = field(default_factory=set) + manual_proofs_before_clear: set[str] = field(default_factory=set) + active_owners: set[WorkflowMode] = field(default_factory=set) + background_owners: set[WorkflowMode] = field(default_factory=set) + pending_child_tasks: int = 0 + stale_child_outputs_fenced: bool | None = None + stale_child_outputs_accepted: int = 0 + truncation_failures: int = 0 + provider_pause_from_truncation: int = 0 + hosted_desktop_route_attempted: bool = False + hosted_desktop_route_unavailable: bool | None = None + hosted_route_attempts: set[str] = field(default_factory=set) + hosted_route_unavailable: set[str] = field(default_factory=set) + active_state_cleared: bool = False + history_preserved: bool | None = None + completed_brainstorm_generated_paper: bool = False + replayed_completed_brainstorm_handoff: bool = False + pruned_papers: set[str] = field(default_factory=set) + model_context_papers: set[str] = field(default_factory=set) + runtime_root_violations: int = 0 + resolved_runtime_paths: list[Path] = field(default_factory=list) + prompt_user_direct_injected: bool | None = None + validator_prompt_has_assistant_memory: bool | None = None + proof_source_context_required: bool | None = None + proof_source_context_present: bool | None = None + generated_appendices_stripped: bool | None = None + direct_sources_excluded_from_rag: bool | None = None + mandatory_source_overflow_visible: bool | None = None + observed_invariants: set[str] = field(default_factory=set) + exercise_observations: set[str] = field(default_factory=set) + persisted_events: list[WorkflowEvent] = field(default_factory=list) + terminal_stop_events: int = 0 + checkpoint: dict[str, object] = field(default_factory=dict) + events: list[WorkflowEvent] = field(default_factory=list) + replay: list[str] = field(default_factory=list) + + def __post_init__(self) -> None: + if self.mode is not WorkflowMode.NONE and not self.active_owners: + self.active_owners.add(self.mode) + + def record(self, action: str, **payload: object) -> None: + if payload: + details = ", ".join(f"{key}={value!r}" for key, value in sorted(payload.items())) + self.replay.append(f"{action}({details})") + else: + self.replay.append(f"{action}()") + + def emit(self, event_type: str, **payload: object) -> None: + self.events.append(WorkflowEvent(event_type=event_type, payload=payload)) + + def ingest_event(self, event_type: str, payload: dict[str, object]) -> None: + self.emit(event_type, **payload) + if event_type == "proof_verified": + proof_id = str(payload.get("proof_id") or "") + if proof_id: + if payload.get("scope") == "manual": + self.manual_proofs_active.add(proof_id) + else: + self.autonomous_proofs.add(proof_id) + elif event_type == "autonomous_proof_provider_paused": + self.previous_phase = self.phase + self.phase = WorkflowPhase.PAUSED + self.provider.exhaust_credit() + self.checkpoint.update( + { + "paused": True, + "phase": self.phase.value, + "resume_phase": self.previous_phase.value, + "source_type": payload.get("source_type"), + "source_id": payload.get("source_id"), + "trigger": payload.get("trigger"), + "candidate_cursor": payload.get("candidate_cursor", 0), + "proof_round": payload.get("proof_round", 1), + } + ) + elif event_type == "autonomous_proof_provider_resumed": + resume_phase = self.checkpoint.get("resume_phase") + if isinstance(resume_phase, str): + self.phase = WorkflowPhase(resume_phase) + self.provider.reset_credit() + self.checkpoint["paused"] = False + self.checkpoint["phase"] = self.phase.value + elif event_type == "research_papers_disabled_brainstorm_complete": + self.phase = WorkflowPhase.TOPIC_EXPLORATION + self.checkpoint["phase"] = self.phase.value diff --git a/tests/workflow_harness/real_adapters/source_tagged_rag.py b/tests/workflow_harness/real_adapters/source_tagged_rag.py new file mode 100644 index 0000000..9e953ad --- /dev/null +++ b/tests/workflow_harness/real_adapters/source_tagged_rag.py @@ -0,0 +1,93 @@ +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Iterable + +from backend.shared.models import ContextPack +from backend.shared.utils import count_tokens + + +@dataclass(frozen=True) +class SourceTaggedDocument: + source: str + text: str + + +@dataclass +class SourceTaggedRagIndex: + """Small deterministic RAG fake that preserves source exclusion semantics.""" + + documents: list[SourceTaggedDocument] = field(default_factory=list) + calls: list[dict[str, object]] = field(default_factory=list) + + def add(self, source: str, text: str) -> None: + self.documents.append(SourceTaggedDocument(source=source, text=text)) + + async def retrieve( + self, + *, + query: str, + max_tokens: int, + exclude_sources: Iterable[str] | None = None, + chunk_size: int | None = None, + **_: object, + ) -> ContextPack: + excluded = set(exclude_sources or ()) + self.calls.append( + { + "query": query, + "max_tokens": max_tokens, + "chunk_size": chunk_size, + "exclude_sources": tuple(sorted(excluded)), + } + ) + + selected: list[SourceTaggedDocument] = [] + used_tokens = 0 + for document in self.documents: + if document.source in excluded: + continue + rendered = f"[SOURCE: {document.source}]\n{document.text}" + document_tokens = count_tokens(rendered) + if selected and used_tokens + document_tokens > max_tokens: + continue + if not selected and document_tokens > max_tokens: + continue + selected.append(document) + used_tokens += document_tokens + + text = "\n\n".join( + f"[SOURCE: {document.source}]\n{document.text}" for document in selected + ) + return ContextPack( + text=text, + evidence=[ + {"source": document.source, "text": document.text} + for document in selected + ], + source_map={ + str(index): document.source + for index, document in enumerate(selected) + }, + coverage=1.0 if selected else 0.0, + answerability=1.0 if selected else 0.0, + ) + + async def retrieve_for_mode( + self, + *, + query: str, + mode: str, + max_tokens: int | None = None, + exclude_sources: Iterable[str] | None = None, + role_context_window: int | None = None, + role_max_output_tokens: int | None = None, + **kwargs: object, + ) -> ContextPack: + del mode, role_context_window, role_max_output_tokens + return await self.retrieve( + query=query, + max_tokens=max_tokens or 100_000, + exclude_sources=exclude_sources, + **kwargs, + ) diff --git a/tests/workflow_harness/recombinatory_generator.py b/tests/workflow_harness/recombinatory_generator.py new file mode 100644 index 0000000..fe5b298 --- /dev/null +++ b/tests/workflow_harness/recombinatory_generator.py @@ -0,0 +1,1199 @@ +from __future__ import annotations + +import hashlib +import json +from copy import deepcopy +from collections import deque +from collections.abc import Callable +from dataclasses import dataclass +from enum import Enum +from pathlib import Path + +from . import actions +from .exercise_evidence import KNOWN_EXERCISE_TOKENS, observed_exercise_tokens +from .invariant_catalog import ( + FIRST_BUILD_FIELDS, + INVARIANTS_BY_ID, + INVARIANT_CATALOG, +) +from .model import WorkflowMode, WorkflowModel, WorkflowPhase + + +Eligibility = Callable[[WorkflowModel], bool] +FailurePredicate = Callable[[WorkflowModel], bool] +ModelFactory = Callable[[], WorkflowModel] + + +@dataclass(frozen=True) +class ActionSpec: + action_id: str + execute: actions.WorkflowAction + fields: tuple[str, ...] + eligible_when: Eligibility + weight: int = 1 + advances: tuple[str, ...] = () + produces: tuple[str, ...] = () + + +@dataclass(frozen=True) +class GenerationTarget: + target_id: str + fields: tuple[str, ...] + invariants: tuple[str, ...] + must_exercise: tuple[str, ...] + max_steps: int = 18 + + +@dataclass(frozen=True) +class GeneratedRun: + target_id: str + seed: int + fields: tuple[str, ...] + observed_fields: frozenset[str] + invariants: tuple[str, ...] + exercised_invariants: frozenset[str] + action_ids: tuple[str, ...] + replay: tuple[str, ...] + observed_evidence: frozenset[str] + final_mode: WorkflowMode + final_phase: WorkflowPhase + + @property + def action_count(self) -> int: + return len(self.action_ids) + + +class GeneratedRunFailure(AssertionError): + pass + + +@dataclass(frozen=True) +class InvariantFailure(Exception): + invariant_id: str + detail: str + + +class ReplayRejected(ValueError): + """The requested replay is not a canonical executable action sequence.""" + + +REPLAY_NOT_REPRODUCED = "replay.not_reproduced" + + +@dataclass(frozen=True) +class ReplayResult: + action_ids: tuple[str, ...] + replay: tuple[str, ...] + failure_category: str | None = None + failure_detail: str = "" + + @property + def failed(self) -> bool: + return ( + self.failure_category is not None + and self.failure_category != REPLAY_NOT_REPRODUCED + ) + + +@dataclass(frozen=True) +class ReplayReduction: + original_action_ids: tuple[str, ...] + shortest_failing_prefix: tuple[str, ...] + action_ids: tuple[str, ...] + failure_category: str | None + + @property + def minimized(self) -> bool: + return self.action_ids != self.original_action_ids + + +INVARIANT_ACTIVATION_EVIDENCE: dict[str, frozenset[str]] = { + "runtime.single_active_workflow": frozenset({"start_blocked"}), + "outputs.at_least_one_output_enabled": frozenset({"no_outputs_rejected"}), + "outputs.proofs_only_no_paper_phase": frozenset({"provider_pause"}), + "outputs.papers_only_skips_proof_work": frozenset({"papers_only"}), + "provider.pause_preserves_checkpoint": frozenset({"provider_pause"}), + "provider.stop_resume_preserves_pause": frozenset({"stop_resume"}), + "provider.reset_wakes_without_corrupting_checkpoint": frozenset({"reset_credit"}), + "assistant.no_validator_injection": frozenset({"prompt_context"}), + "assistant.disable_clears_live_pack": frozenset({"assistant_disable"}), + "proof_scope.manual_not_in_autonomous_current": frozenset({"manual_proof"}), + "proof_scope.manual_clear_archives_active": frozenset({"manual_archive"}), + "state.clear_removes_active_preserves_history": frozenset({"manual_archive"}), + "events.frontend_events_include_scope_phase": frozenset({"scoped_event"}), + "events.proof_verified_after_registration": frozenset({"registered_proof_event"}), + "proof_scope.autonomous_events_not_manual": frozenset({"registered_proof_event"}), + "proof_runtime.no_lean_when_disabled": frozenset({"papers_only"}), + "proof_runtime.no_smt_when_disabled": frozenset({"papers_only"}), + "prompt.validator_excludes_assistant_memory": frozenset({"prompt_context"}), + "events.context_overflow_route_identity": frozenset({"context_overflow"}), + "events.context_overflow_persists_across_reload": frozenset({"context_overflow"}), + "events.context_overflow_terminal_stop_once": frozenset({"context_overflow"}), + "events.proof_context_overflow_nonfatal": frozenset({"context_overflow"}), + "prompt.user_prompt_direct_injected": frozenset({"prompt_context"}), + "prompt.proof_source_context_required": frozenset({"prompt_context"}), + "prompt.direct_sources_excluded_from_rag": frozenset({"rag_source_exclusion"}), + "prompt.mandatory_source_overflow_fails_visible": frozenset({"mandatory_source_overflow"}), + "prompt.generated_appendices_stripped": frozenset({"prompt_context"}), +} + + +def _idle(model: WorkflowModel) -> bool: + return model.mode is WorkflowMode.NONE and not model.checkpoint + + +def _active(model: WorkflowModel) -> bool: + return model.mode is not WorkflowMode.NONE + + +def _autonomous_phase(phase: WorkflowPhase) -> Eligibility: + return lambda model: model.mode is WorkflowMode.AUTONOMOUS and model.phase is phase + + +def _manual_writing(model: WorkflowModel) -> bool: + return ( + model.mode is WorkflowMode.MANUAL_COMPILER + and model.phase is WorkflowPhase.PAPER_WRITING + and not model.lean.enabled + ) + + +def _manual_proof_ready(model: WorkflowModel) -> bool: + return model.mode is WorkflowMode.MANUAL_COMPILER and model.lean.enabled + + +def _manual_aggregating(model: WorkflowModel) -> bool: + return ( + model.mode is WorkflowMode.MANUAL_AGGREGATOR + and model.phase is WorkflowPhase.MANUAL_AGGREGATION + ) + + +def _leanoj_phase(phase: WorkflowPhase) -> Eligibility: + return lambda model: model.mode is WorkflowMode.LEANOJ and model.phase is phase + + +def _leanoj_active(model: WorkflowModel) -> bool: + return model.mode is WorkflowMode.LEANOJ + + +def _can_stop(model: WorkflowModel) -> bool: + return _active(model) + + +def _can_resume(model: WorkflowModel) -> bool: + return model.mode is WorkflowMode.NONE and bool(model.checkpoint) + + +def _can_reset_credit(model: WorkflowModel) -> bool: + return ( + model.mode is not WorkflowMode.NONE + and model.phase is WorkflowPhase.PAUSED + and model.provider.credit_exhausted + ) + + +def _can_disable_assistant(model: WorkflowModel) -> bool: + return model.assistant.enabled and bool(model.assistant.live_pack) + + +def _can_refresh_assistant(model: WorkflowModel) -> bool: + return _active(model) and model.assistant.enabled and not model.assistant.live_pack + + +def _can_prepare_prompt_context(model: WorkflowModel) -> bool: + return _active(model) and model.assistant.enabled + + +def _can_clear_manual(model: WorkflowModel) -> bool: + return ( + model.mode is WorkflowMode.MANUAL_COMPILER + and bool(model.manual_proofs_active) + ) + + +def _can_clear_stopped_checkpoint(model: WorkflowModel) -> bool: + return ( + model.mode is WorkflowMode.NONE + and bool(model.checkpoint) + and not model.provider.credit_exhausted + ) + + +ACTION_SPECS: tuple[ActionSpec, ...] = ( + ActionSpec( + "start_autonomous_proofs_only", + actions.start_autonomous_proofs_only, + ("allowed_outputs", "provider_pause_resume", "workflow_filesystem_state"), + _idle, + advances=("provider_pause", "stop_resume", "reset_credit"), + ), + ActionSpec( + "start_autonomous_papers_and_proofs", + actions.start_autonomous_papers_and_proofs, + ("allowed_outputs", "assistant_memory", "prompt_context", "websocket_api_contracts"), + _idle, + advances=( + "assistant_pack", + "assistant_disable", + "prompt_context", + "registered_proof_event", + "scoped_event", + ), + ), + ActionSpec( + "start_autonomous_papers_only", + actions.start_autonomous_papers_only, + ("allowed_outputs", "proof_runtime_gating"), + _idle, + advances=("papers_only",), + ), + ActionSpec( + "start_autonomous_no_outputs", + actions.start_autonomous_no_outputs, + ("allowed_outputs", "proof_runtime_gating"), + _idle, + produces=("no_outputs_rejected",), + ), + ActionSpec( + "start_manual_compiler", + actions.start_manual_compiler, + ("proof_scope_isolation", "workflow_filesystem_state"), + _idle, + advances=("manual_proof", "manual_archive"), + ), + ActionSpec( + "start_manual_aggregator", + actions.start_manual_aggregator, + ("runtime_exclusivity", "workflow_filesystem_state"), + _idle, + advances=("manual_aggregator_accept", "manual_aggregator_clear", "stop_resume"), + ), + ActionSpec( + "accept_manual_aggregator_submission", + actions.accept_manual_aggregator_submission, + ("workflow_filesystem_state", "prompt_context"), + _manual_aggregating, + produces=("manual_aggregator_accept",), + ), + ActionSpec( + "start_leanoj", + actions.start_leanoj, + ("runtime_exclusivity", "workflow_filesystem_state"), + _idle, + advances=("leanoj_draft", "leanoj_skip", "leanoj_force", "stop_resume", "leanoj_clear"), + ), + ActionSpec( + "edit_leanoj_master_proof", + actions.edit_leanoj_master_proof, + ("workflow_filesystem_state", "prompt_context"), + _leanoj_active, + produces=("leanoj_draft",), + ), + ActionSpec( + "skip_leanoj_brainstorm", + actions.skip_leanoj_brainstorm, + ("runtime_exclusivity", "workflow_filesystem_state"), + _leanoj_phase(WorkflowPhase.LEANOJ_BRAINSTORM), + produces=("leanoj_skip",), + advances=("leanoj_force",), + ), + ActionSpec( + "force_leanoj_brainstorm", + actions.force_leanoj_brainstorm, + ("runtime_exclusivity", "workflow_filesystem_state"), + _leanoj_phase(WorkflowPhase.LEANOJ_FINAL), + produces=("leanoj_force",), + ), + ActionSpec( + "attempt_conflicting_start", + actions.start_manual_compiler, + ("runtime_exclusivity", "websocket_api_contracts"), + _active, + produces=("start_blocked",), + ), + ActionSpec( + "complete_topic_exploration", + actions.complete_topic_exploration, + ("allowed_outputs", "provider_pause_resume", "workflow_filesystem_state"), + _autonomous_phase(WorkflowPhase.TOPIC_EXPLORATION), + advances=("provider_pause", "reset_credit", "papers_only"), + ), + ActionSpec( + "simulate_credit_exhaustion", + actions.simulate_credit_exhaustion, + ("provider_pause_resume", "proof_runtime_gating"), + lambda model: ( + model.mode is WorkflowMode.AUTONOMOUS + and model.phase is WorkflowPhase.TIER1_AGGREGATION + and model.allow_mathematical_proofs + and not model.provider.credit_exhausted + ), + weight=20, + advances=("provider_pause", "reset_credit"), + ), + ActionSpec( + "complete_brainstorm", + actions.complete_brainstorm, + ("allowed_outputs", "provider_pause_resume", "proof_runtime_gating"), + _autonomous_phase(WorkflowPhase.TIER1_AGGREGATION), + advances=("provider_pause", "reset_credit", "papers_only"), + ), + ActionSpec( + "force_paper_writing", + actions.force_paper_writing, + ("allowed_outputs", "workflow_filesystem_state"), + _autonomous_phase(WorkflowPhase.TIER1_AGGREGATION), + ), + ActionSpec( + "complete_paper", + actions.complete_paper, + ("allowed_outputs", "proof_runtime_gating", "workflow_filesystem_state"), + _autonomous_phase(WorkflowPhase.PAPER_TITLE), + ), + ActionSpec( + "enter_autonomous_paper_checkpoint", + actions.enter_autonomous_paper_checkpoint, + ("allowed_outputs", "proof_runtime_gating", "workflow_filesystem_state"), + _autonomous_phase(WorkflowPhase.PAPER_TITLE), + advances=("paper_checkpoint",), + ), + ActionSpec( + "complete_autonomous_paper_checkpoint", + actions.complete_autonomous_paper_checkpoint, + ("proof_runtime_gating", "workflow_filesystem_state", "websocket_api_contracts"), + _autonomous_phase(WorkflowPhase.PAPER_PROOF), + produces=("paper_checkpoint",), + ), + ActionSpec( + "attempt_conflict_during_autonomous_paper_checkpoint", + actions.attempt_conflict_during_autonomous_paper_checkpoint, + ("runtime_exclusivity", "websocket_api_contracts"), + _autonomous_phase(WorkflowPhase.PAPER_PROOF), + produces=("start_blocked",), + ), + ActionSpec( + "enable_manual_lean", + actions.enable_manual_lean, + ("proof_runtime_gating", "proof_scope_isolation"), + _manual_writing, + advances=("manual_proof", "manual_archive"), + ), + ActionSpec( + "run_manual_proof_check", + actions.run_manual_proof_check, + ("proof_runtime_gating", "proof_scope_isolation"), + _manual_proof_ready, + advances=("manual_archive",), + produces=("manual_proof",), + ), + ActionSpec( + "refresh_assistant_pack", + actions.refresh_assistant_pack, + ("assistant_memory", "prompt_context", "proof_scope_isolation"), + _can_refresh_assistant, + advances=("assistant_disable", "manual_archive"), + produces=("assistant_pack",), + ), + ActionSpec( + "prepare_prompt_context", + actions.prepare_prompt_context, + ("prompt_context", "assistant_memory", "proof_scope_isolation"), + _can_prepare_prompt_context, + produces=("prompt_context",), + ), + ActionSpec( + "disable_session_history", + actions.disable_session_history, + ("assistant_memory", "prompt_context"), + _can_disable_assistant, + produces=("assistant_disable",), + ), + ActionSpec( + "emit_frontend_scoped_event", + actions.emit_frontend_scoped_event, + ("websocket_api_contracts", "proof_scope_isolation"), + _active, + produces=("scoped_event",), + ), + ActionSpec( + "emit_registered_proof_verified", + actions.emit_registered_proof_verified, + ("websocket_api_contracts", "proof_scope_isolation"), + _active, + produces=("registered_proof_event", "scoped_event"), + ), + ActionSpec( + "emit_context_overflow_contract_events", + actions.emit_context_overflow_contract_events, + ( + "websocket_api_contracts", + "workflow_filesystem_state", + "provider_pause_resume", + "runtime_exclusivity", + "proof_runtime_gating", + ), + _active, + produces=("context_overflow",), + ), + ActionSpec( + "verify_rag_source_exclusion", + actions.verify_rag_source_exclusion, + ("prompt_context", "workflow_filesystem_state"), + _active, + produces=("rag_source_exclusion",), + ), + ActionSpec( + "reject_mandatory_source_overflow", + actions.reject_mandatory_source_overflow, + ("prompt_context", "proof_runtime_gating", "websocket_api_contracts"), + _active, + produces=("mandatory_source_overflow",), + ), + ActionSpec( + "stop", + actions.stop, + ("provider_pause_resume", "workflow_filesystem_state", "runtime_exclusivity"), + _can_stop, + advances=("stop_resume",), + ), + ActionSpec( + "resume", + actions.resume, + ("provider_pause_resume", "workflow_filesystem_state", "runtime_exclusivity"), + _can_resume, + produces=("stop_resume",), + ), + ActionSpec( + "reset_credit", + actions.reset_credit, + ("provider_pause_resume", "workflow_filesystem_state"), + _can_reset_credit, + produces=("reset_credit",), + ), + ActionSpec( + "clear_manual_state", + actions.clear, + ("proof_scope_isolation", "workflow_filesystem_state", "assistant_memory"), + _can_clear_manual, + produces=("manual_archive",), + ), + ActionSpec( + "clear_stopped_checkpoint", + actions.clear, + ("workflow_filesystem_state", "runtime_exclusivity"), + _can_clear_stopped_checkpoint, + ), + ActionSpec( + "clear_manual_aggregator_state", + actions.clear, + ("workflow_filesystem_state", "runtime_exclusivity", "prompt_context"), + lambda model: _manual_aggregating(model) and bool(model.manual_aggregator_submissions), + produces=("manual_aggregator_clear",), + ), + ActionSpec( + "clear_leanoj_state", + actions.clear, + ("workflow_filesystem_state", "runtime_exclusivity", "prompt_context"), + _leanoj_active, + produces=("leanoj_clear",), + ), +) + + +DEFAULT_TARGETS: tuple[GenerationTarget, ...] = ( + GenerationTarget( + target_id="generated_allowed_outputs_provider_pause_resume", + fields=("allowed_outputs", "provider_pause_resume", "workflow_filesystem_state"), + invariants=( + "outputs.proofs_only_no_paper_phase", + "provider.pause_preserves_checkpoint", + "provider.stop_resume_preserves_pause", + "provider.reset_wakes_without_corrupting_checkpoint", + ), + must_exercise=("provider_pause", "stop_resume", "reset_credit"), + max_steps=16, + ), + GenerationTarget( + target_id="generated_assistant_prompt_context", + fields=("assistant_memory", "prompt_context"), + invariants=( + "assistant.disable_clears_live_pack", + "assistant.no_validator_injection", + "prompt.validator_excludes_assistant_memory", + ), + must_exercise=("assistant_pack", "prompt_context", "assistant_disable"), + max_steps=12, + ), + GenerationTarget( + target_id="generated_manual_scope_clear_archive", + fields=("proof_scope_isolation", "workflow_filesystem_state", "assistant_memory"), + invariants=( + "proof_scope.manual_not_in_autonomous_current", + "proof_scope.manual_clear_archives_active", + "state.clear_removes_active_preserves_history", + ), + must_exercise=("manual_proof", "assistant_pack", "manual_archive"), + max_steps=14, + ), + GenerationTarget( + target_id="generated_scoped_registered_proof_events", + fields=("websocket_api_contracts", "proof_scope_isolation"), + invariants=( + "events.frontend_events_include_scope_phase", + "events.proof_verified_after_registration", + "proof_scope.autonomous_events_not_manual", + ), + must_exercise=("scoped_event", "registered_proof_event"), + max_steps=10, + ), + GenerationTarget( + target_id="generated_output_and_proof_runtime_gating", + fields=("proof_runtime_gating", "allowed_outputs"), + invariants=( + "outputs.at_least_one_output_enabled", + "outputs.papers_only_skips_proof_work", + "proof_runtime.no_lean_when_disabled", + "proof_runtime.no_smt_when_disabled", + ), + must_exercise=("no_outputs_rejected", "papers_only"), + max_steps=10, + ), + GenerationTarget( + target_id="generated_runtime_start_exclusivity", + fields=("runtime_exclusivity", "websocket_api_contracts"), + invariants=("runtime.single_active_workflow",), + must_exercise=("start_blocked",), + max_steps=8, + ), + GenerationTarget( + target_id="generated_context_overflow_contract", + fields=( + "websocket_api_contracts", + "workflow_filesystem_state", + "provider_pause_resume", + "runtime_exclusivity", + "proof_runtime_gating", + ), + invariants=( + "events.context_overflow_route_identity", + "events.context_overflow_persists_across_reload", + "events.context_overflow_terminal_stop_once", + "events.proof_context_overflow_nonfatal", + ), + must_exercise=("context_overflow",), + max_steps=6, + ), + GenerationTarget( + target_id="generated_rag_prompt_boundaries", + fields=( + "prompt_context", + "workflow_filesystem_state", + "proof_runtime_gating", + "websocket_api_contracts", + ), + invariants=( + "prompt.user_prompt_direct_injected", + "prompt.proof_source_context_required", + "prompt.direct_sources_excluded_from_rag", + "prompt.mandatory_source_overflow_fails_visible", + "prompt.generated_appendices_stripped", + ), + must_exercise=( + "prompt_context", + "rag_source_exclusion", + "mandatory_source_overflow", + ), + max_steps=8, + ), + GenerationTarget( + target_id="generated_manual_aggregator_lifecycle", + fields=("runtime_exclusivity", "workflow_filesystem_state"), + invariants=("runtime.single_active_workflow",), + must_exercise=( + "manual_aggregator_accept", + "manual_aggregator_clear", + "start_blocked", + ), + max_steps=6, + ), + GenerationTarget( + target_id="generated_leanoj_durable_lifecycle", + fields=("runtime_exclusivity", "workflow_filesystem_state"), + invariants=( + "runtime.single_active_workflow", + "provider.stop_resume_preserves_pause", + ), + must_exercise=( + "leanoj_draft", + "leanoj_skip", + "leanoj_force", + "stop_resume", + "leanoj_clear", + "start_blocked", + ), + max_steps=10, + ), + GenerationTarget( + target_id="generated_autonomous_paper_checkpoint", + fields=( + "runtime_exclusivity", + "workflow_filesystem_state", + "websocket_api_contracts", + ), + invariants=("runtime.single_active_workflow",), + must_exercise=("paper_checkpoint", "start_blocked"), + max_steps=7, + ), +) + + +def validate_action_specs(action_specs: tuple[ActionSpec, ...] = ACTION_SPECS) -> None: + action_ids = [spec.action_id for spec in action_specs] + if len(action_ids) != len(set(action_ids)): + raise AssertionError("Action registry contains duplicate action IDs.") + for spec in action_specs: + if not spec.action_id or not callable(spec.execute) or not callable(spec.eligible_when): + raise AssertionError(f"Invalid action descriptor: {spec!r}.") + if spec.weight <= 0: + raise AssertionError(f"{spec.action_id} must have a positive weight.") + unknown_fields = set(spec.fields) - FIRST_BUILD_FIELDS + if unknown_fields: + raise AssertionError( + f"{spec.action_id} references unknown fields {sorted(unknown_fields)!r}." + ) + unknown_evidence = set(spec.advances).union(spec.produces) - KNOWN_EXERCISE_TOKENS + if unknown_evidence: + raise AssertionError( + f"{spec.action_id} references unknown evidence {sorted(unknown_evidence)!r}." + ) + + +def action_specs_by_id( + action_specs: tuple[ActionSpec, ...] = ACTION_SPECS, +) -> dict[str, ActionSpec]: + """Return the validated canonical action registry keyed by stable action ID.""" + validate_action_specs(action_specs) + return {spec.action_id: spec for spec in action_specs} + + +def _failure_from_model( + model: WorkflowModel, + *, + failure_predicate: FailurePredicate | None, + failure_category: str | None, +) -> tuple[str, str] | None: + try: + _assert_all_invariants_with_id(model) + except InvariantFailure as exc: + return exc.invariant_id, exc.detail + if failure_predicate is not None and failure_predicate(model): + return failure_category or "replay.failure_predicate", "Failure predicate matched." + return None + + +def replay_action_ids( + model: WorkflowModel, + action_ids: tuple[str, ...] | list[str], + *, + action_specs: tuple[ActionSpec, ...] = ACTION_SPECS, + failure_predicate: FailurePredicate | None = None, + failure_category: str | None = None, +) -> ReplayResult: + """Replay stable IDs, rejecting unknown or state-ineligible actions.""" + registry = action_specs_by_id(action_specs) + canonical_ids = tuple(action_ids) + for index, action_id in enumerate(canonical_ids): + spec = registry.get(action_id) + if spec is None: + raise ReplayRejected(f"Unknown action ID {action_id!r} at index {index}.") + if not spec.eligible_when(model): + raise ReplayRejected( + f"Action {action_id!r} is ineligible at index {index} " + f"for mode={model.mode.value}, phase={model.phase.value}." + ) + replay_before = len(model.replay) + spec.execute(model) + if len(model.replay) <= replay_before: + raise ReplayRejected( + f"Action {action_id!r} at index {index} did not record its execution." + ) + failure = _failure_from_model( + model, + failure_predicate=failure_predicate, + failure_category=failure_category, + ) + if failure is not None: + category, detail = failure + if failure_category is not None and category != failure_category: + return ReplayResult( + action_ids=canonical_ids[: index + 1], + replay=tuple(model.replay), + failure_category=REPLAY_NOT_REPRODUCED, + failure_detail=( + f"Requested failure category {failure_category!r} was not " + f"reproduced; observed {category!r}: {detail}" + ), + ) + return ReplayResult( + action_ids=canonical_ids[: index + 1], + replay=tuple(model.replay), + failure_category=category, + failure_detail=detail, + ) + if failure_category is not None: + return ReplayResult( + action_ids=canonical_ids, + replay=tuple(model.replay), + failure_category=REPLAY_NOT_REPRODUCED, + failure_detail=( + f"Requested failure category {failure_category!r} was not reproduced." + ), + ) + return ReplayResult(action_ids=canonical_ids, replay=tuple(model.replay)) + + +def shortest_failing_prefix( + model_factory: ModelFactory, + action_ids: tuple[str, ...] | list[str], + *, + action_specs: tuple[ActionSpec, ...] = ACTION_SPECS, + failure_predicate: FailurePredicate | None = None, + failure_category: str | None = None, +) -> tuple[str, ...]: + """Return the first prefix reproducing the requested failure category.""" + canonical_ids = tuple(action_ids) + for size in range(1, len(canonical_ids) + 1): + result = replay_action_ids( + model_factory(), + canonical_ids[:size], + action_specs=action_specs, + failure_predicate=failure_predicate, + failure_category=failure_category, + ) + if result.failed and result.failure_category != REPLAY_NOT_REPRODUCED and ( + failure_category is None or result.failure_category == failure_category + ): + return canonical_ids[:size] + return canonical_ids + + +def reduce_failing_action_ids( + model_factory: ModelFactory, + action_ids: tuple[str, ...] | list[str], + *, + action_specs: tuple[ActionSpec, ...] = ACTION_SPECS, + failure_predicate: FailurePredicate | None = None, + failure_category: str | None = None, +) -> ReplayReduction: + """Minimize a reproducing replay via shortest-prefix then chunk deletion.""" + original = tuple(action_ids) + prefix = shortest_failing_prefix( + model_factory, + original, + action_specs=action_specs, + failure_predicate=failure_predicate, + failure_category=failure_category, + ) + baseline = replay_action_ids( + model_factory(), + prefix, + action_specs=action_specs, + failure_predicate=failure_predicate, + failure_category=failure_category, + ) + if not baseline.failed or baseline.failure_category == REPLAY_NOT_REPRODUCED: + return ReplayReduction(original, prefix, original, None) + + category = failure_category or baseline.failure_category + + def reproduces(candidate: tuple[str, ...]) -> bool: + if not candidate: + return False + try: + result = replay_action_ids( + model_factory(), + candidate, + action_specs=action_specs, + failure_predicate=failure_predicate, + failure_category=failure_category, + ) + except ReplayRejected: + return False + return result.failed and result.failure_category == category + + reduced = prefix + granularity = 2 + while len(reduced) >= 2: + chunk_size = max(1, (len(reduced) + granularity - 1) // granularity) + removed = False + for start in range(0, len(reduced), chunk_size): + candidate = reduced[:start] + reduced[start + chunk_size :] + if reproduces(candidate): + reduced = candidate + granularity = max(2, granularity - 1) + removed = True + break + if removed: + continue + if granularity >= len(reduced): + break + granularity = min(len(reduced), granularity * 2) + + return ReplayReduction(original, prefix, reduced, category) + + +def validate_generation_target(target: GenerationTarget) -> None: + if not target.target_id: + raise AssertionError("Generation target is missing target_id.") + if len(target.fields) < 2 or len(set(target.fields)) != len(target.fields): + raise AssertionError(f"{target.target_id} must declare at least two unique fields.") + unknown_fields = set(target.fields) - FIRST_BUILD_FIELDS + if unknown_fields: + raise AssertionError( + f"{target.target_id} references unknown fields {sorted(unknown_fields)!r}." + ) + if not target.invariants: + raise AssertionError(f"{target.target_id} does not declare invariants.") + if len(set(target.invariants)) != len(target.invariants): + raise AssertionError(f"{target.target_id} declares duplicate invariants.") + if len(set(target.must_exercise)) != len(target.must_exercise): + raise AssertionError(f"{target.target_id} declares duplicate evidence tokens.") + + connected_fields: set[str] = set() + for invariant_id in target.invariants: + invariant = INVARIANTS_BY_ID.get(invariant_id) + if invariant is None: + raise AssertionError( + f"{target.target_id} references unknown invariant {invariant_id!r}." + ) + if "model" not in invariant.adapters: + raise AssertionError( + f"{target.target_id} invariant {invariant_id!r} does not support model adapter." + ) + connected_fields.add(invariant.field) + connected_fields.update(invariant.crossed_fields) + if invariant_id not in INVARIANT_ACTIVATION_EVIDENCE: + raise AssertionError( + f"{target.target_id} invariant {invariant_id!r} has no activation evidence." + ) + unmatched_fields = set(target.fields) - connected_fields + if unmatched_fields: + raise AssertionError( + f"{target.target_id} fields are not connected to its invariants: " + f"{sorted(unmatched_fields)!r}." + ) + unknown_evidence = set(target.must_exercise) - KNOWN_EXERCISE_TOKENS + if unknown_evidence: + raise AssertionError( + f"{target.target_id} references unknown evidence {sorted(unknown_evidence)!r}." + ) + if target.max_steps <= 0: + raise AssertionError(f"{target.target_id} max_steps must be positive.") + + +def eligible_action_specs( + model: WorkflowModel, + action_specs: tuple[ActionSpec, ...] = ACTION_SPECS, +) -> tuple[ActionSpec, ...]: + return tuple(spec for spec in action_specs if spec.eligible_when(model)) + + +def _weight_for( + spec: ActionSpec, + target: GenerationTarget, + missing_evidence: frozenset[str], +) -> int: + field_overlap = len(set(spec.fields).intersection(target.fields)) + evidence_overlap = len( + set(spec.advances).union(spec.produces).intersection(missing_evidence) + ) + return spec.weight + (8 * field_overlap) + (128 * evidence_overlap) + + +def _signature_value(value: object) -> object: + if isinstance(value, Enum): + return value.value + if isinstance(value, Path): + return "<runtime-root>" + if isinstance(value, dict): + return tuple( + (str(key), _signature_value(item)) + for key, item in sorted(value.items(), key=lambda pair: str(pair[0])) + ) + if isinstance(value, (list, tuple)): + return tuple(_signature_value(item) for item in value) + if isinstance(value, (set, frozenset)): + return tuple(sorted((_signature_value(item) for item in value), key=repr)) + if hasattr(value, "__dataclass_fields__"): + ignored = {"runtime_root", "replay"} + return tuple( + (name, _signature_value(getattr(value, name))) + for name in value.__dataclass_fields__ + if name not in ignored + ) + return value + + +def _planning_signature( + model: WorkflowModel, + observed: frozenset[str], + observed_fields: frozenset[str], +) -> tuple[object, frozenset[str], frozenset[str]]: + return (_signature_value(model), observed, observed_fields) + + +def _seeded_action_order( + specs: tuple[ActionSpec, ...], + *, + target: GenerationTarget, + seed: int, + signature: object, + missing_evidence: frozenset[str], +) -> tuple[ActionSpec, ...]: + canonical_state = json.dumps( + _signature_value(signature), + ensure_ascii=True, + separators=(",", ":"), + sort_keys=True, + ) + + def tie_break(spec: ActionSpec) -> str: + material = ( + f"{seed}\0{target.target_id}\0{canonical_state}\0" + f"{','.join(sorted(missing_evidence))}\0{spec.action_id}" + ) + return hashlib.sha256(material.encode("utf-8")).hexdigest() + + return tuple( + sorted( + specs, + key=lambda spec: ( + -_weight_for(spec, target, missing_evidence), + tie_break(spec), + spec.action_id, + ), + ) + ) + + +def _plan_reachable_path( + model: WorkflowModel, + target: GenerationTarget, + *, + seed: int, + action_specs: tuple[ActionSpec, ...], + observed: frozenset[str], + observed_fields: frozenset[str], + remaining_steps: int, +) -> tuple[ActionSpec, ...]: + required_evidence = frozenset(target.must_exercise) + required_fields = frozenset(target.fields) + frontier: deque[ + tuple[WorkflowModel, frozenset[str], frozenset[str], tuple[ActionSpec, ...]] + ] = deque([(deepcopy(model), observed, observed_fields, ())]) + visited = {_planning_signature(frontier[0][0], observed, observed_fields)} + reachable_evidence = set(observed) + reachable_fields = set(observed_fields) + + while frontier: + state, state_observed, state_fields, path = frontier.popleft() + if ( + required_evidence.issubset(state_observed) + and required_fields.issubset(state_fields) + ): + return path + if len(path) >= remaining_steps: + continue + + missing = required_evidence - state_observed + eligible = eligible_action_specs(state, action_specs) + progress_actions = tuple( + spec + for spec in eligible + if set(spec.advances).union(spec.produces).intersection(missing) + ) + if progress_actions: + eligible = progress_actions + ordered = _seeded_action_order( + eligible, + target=target, + seed=seed, + signature=_planning_signature(state, state_observed, state_fields), + missing_evidence=missing, + ) + for spec in ordered: + candidate = deepcopy(state) + replay_before = len(candidate.replay) + spec.execute(candidate) + if len(candidate.replay) <= replay_before: + continue + try: + _assert_all_invariants_with_id(candidate) + except InvariantFailure: + continue + next_observed = frozenset( + set(state_observed).union(observed_exercise_tokens(candidate)) + ) + next_fields = frozenset(set(state_fields).union(spec.fields)) + reachable_evidence.update(next_observed) + reachable_fields.update(next_fields) + signature = _planning_signature(candidate, next_observed, next_fields) + if signature in visited: + continue + visited.add(signature) + frontier.append((candidate, next_observed, next_fields, path + (spec,))) + + missing_evidence = sorted(required_evidence - reachable_evidence) + missing_fields = sorted(required_fields - reachable_fields) + raise GeneratedRunFailure( + format_failure( + seed=seed, + fields=target.fields, + action_count=0, + invariant="generator.target_unreachable", + replay=model.replay, + detail=( + f"No path within {remaining_steps} steps; " + f"missing evidence {missing_evidence!r}; " + f"missing fields {missing_fields!r}; " + f"visited {len(visited)} state/evidence signatures." + ), + ) + ) + + +def _assert_all_invariants_with_id(model: WorkflowModel) -> None: + for invariant in INVARIANT_CATALOG: + try: + invariant.checker(model) + except AssertionError as exc: + raise InvariantFailure(invariant.invariant_id, str(exc)) from exc + + +def exercised_target_invariants( + target: GenerationTarget, + observed_evidence: set[str] | frozenset[str], +) -> frozenset[str]: + return frozenset( + invariant_id + for invariant_id in target.invariants + if INVARIANT_ACTIVATION_EVIDENCE[invariant_id].issubset(observed_evidence) + ) + + +def format_failure( + *, + seed: int, + fields: tuple[str, ...], + action_count: int, + invariant: str, + replay: tuple[str, ...] | list[str], + detail: str, +) -> str: + numbered_replay = "\n".join( + f"{index}. {action}" for index, action in enumerate(replay, start=1) + ) + return ( + f"Seed: {seed}\n" + f"Fields: {' x '.join(fields) if fields else '(unknown)'}\n" + f"Action count: {action_count}\n" + f"Invariant: {invariant}\n" + f"Detail: {detail}\n" + f"Replay:\n{numbered_replay or '(empty)'}" + ) + + +def run_generated_scenario( + model: WorkflowModel, + target: GenerationTarget, + *, + seed: int, + action_specs: tuple[ActionSpec, ...] = ACTION_SPECS, +) -> GeneratedRun: + validate_action_specs(action_specs) + validate_generation_target(target) + selected_action_ids: list[str] = [] + observed: set[str] = set() + observed_fields: set[str] = set() + + try: + plan = _plan_reachable_path( + model, + target, + seed=seed, + action_specs=action_specs, + observed=frozenset(), + observed_fields=frozenset(), + remaining_steps=target.max_steps, + ) + except GeneratedRunFailure as exc: + raise + + for chosen in plan: + replay_before = len(model.replay) + chosen.execute(model) + selected_action_ids.append(chosen.action_id) + if len(model.replay) <= replay_before: + raise GeneratedRunFailure( + format_failure( + seed=seed, + fields=target.fields, + action_count=len(selected_action_ids), + invariant="generator.action_records_replay", + replay=model.replay, + detail=f"Action {chosen.action_id!r} did not record its execution.", + ) + ) + + try: + _assert_all_invariants_with_id(model) + except InvariantFailure as exc: + raise GeneratedRunFailure( + format_failure( + seed=seed, + fields=target.fields, + action_count=len(selected_action_ids), + invariant=exc.invariant_id, + replay=model.replay, + detail=exc.detail, + ) + ) from exc + observed_fields.update(chosen.fields) + observed.update(observed_exercise_tokens(model)) + + missing = frozenset(target.must_exercise) - observed + missing_fields = frozenset(target.fields) - observed_fields + exercised_invariants = exercised_target_invariants(target, observed) + missing_invariants = frozenset(target.invariants) - exercised_invariants + if missing or missing_fields or missing_invariants: + raise GeneratedRunFailure( + format_failure( + seed=seed, + fields=target.fields, + action_count=len(selected_action_ids), + invariant="coverage.required_evidence", + replay=model.replay, + detail=( + f"Missing evidence {sorted(missing)!r}; " + f"missing fields {sorted(missing_fields)!r}; " + f"unexercised invariants {sorted(missing_invariants)!r}; " + f"observed evidence {sorted(observed)!r}; " + f"observed fields {sorted(observed_fields)!r}." + ), + ) + ) + + return GeneratedRun( + target_id=target.target_id, + seed=seed, + fields=target.fields, + observed_fields=frozenset(observed_fields), + invariants=target.invariants, + exercised_invariants=exercised_invariants, + action_ids=tuple(selected_action_ids), + replay=tuple(model.replay), + observed_evidence=frozenset(observed), + final_mode=model.mode, + final_phase=model.phase, + ) diff --git a/tests/workflow_real_adapters/__init__.py b/tests/workflow_real_adapters/__init__.py new file mode 100644 index 0000000..25d0c3a --- /dev/null +++ b/tests/workflow_real_adapters/__init__.py @@ -0,0 +1 @@ +"""Real-code workflow adapter tests.""" diff --git a/tests/workflow_real_adapters/coverage_records.py b/tests/workflow_real_adapters/coverage_records.py new file mode 100644 index 0000000..6c0dc44 --- /dev/null +++ b/tests/workflow_real_adapters/coverage_records.py @@ -0,0 +1,423 @@ +from __future__ import annotations + +from dataclasses import replace + +from tests.workflow_harness.coverage_metadata import InteractionCoverage + + +def _node(file_name: str, test_name: str) -> str: + return f"tests/workflow_real_adapters/{file_name}::{test_name}" + + +PROOF_ROUTE_COVERAGE = ( + InteractionCoverage( + scenario_id="real_manual_compiler_proof_only_uses_rigor_settings_and_manual_scope", + fields=("allowed_outputs", "proof_scope_isolation", "prompt_context"), + invariants=("proof_scope.manual_not_in_autonomous_current",), + adapter="real_coordinator", + result="passed", + test_file="tests/workflow_real_adapters/test_proof_routes.py", + evidence=("manual_scope", "rigor_settings"), + ), + InteractionCoverage( + scenario_id="real_autonomous_proofs_only_handoff_returns_to_topic_exploration", + fields=("allowed_outputs", "proof_runtime_gating", "workflow_filesystem_state"), + invariants=("outputs.proofs_only_no_paper_phase",), + adapter="real_coordinator", + result="passed", + test_file="tests/workflow_real_adapters/test_proof_routes.py", + evidence=("proofs_only_handoff", "topic_exploration"), + ), + InteractionCoverage( + scenario_id="real_autonomous_provider_credit_pause_preserves_checkpoint_and_resumes", + fields=("provider_pause_resume", "workflow_filesystem_state", "proof_runtime_gating"), + invariants=("provider.pause_preserves_checkpoint",), + adapter="real_coordinator", + result="passed", + test_file="tests/workflow_real_adapters/test_proof_routes.py", + evidence=("provider_pause", "checkpoint_resume"), + ), + InteractionCoverage( + scenario_id="real_leanoj_master_proof_stop_resume_preserves_isolated_state", + fields=("workflow_filesystem_state", "proof_scope_isolation", "runtime_exclusivity"), + invariants=("proof_scope.autonomous_events_not_manual",), + adapter="real_coordinator", + result="passed", + test_file="tests/workflow_real_adapters/test_proof_routes.py", + evidence=("master_proof_persisted", "session_isolation"), + ), +) + +ROUTE_CONFLICT_COVERAGE = ( + InteractionCoverage( + scenario_id="real_route_start_conflicts_preserve_single_workflow", + fields=("runtime_exclusivity", "websocket_api_contracts"), + invariants=("runtime.single_active_workflow",), + adapter="real_route", + result="passed", + test_file="tests/workflow_real_adapters/test_route_conflicts.py", + evidence=("route_conflict_response", "single_active_workflow"), + ), + InteractionCoverage( + scenario_id="real_route_pending_child_activity_counts_as_active", + fields=("runtime_exclusivity", "workflow_filesystem_state"), + invariants=("runtime.single_active_workflow",), + adapter="real_route", + result="passed", + test_file="tests/workflow_real_adapters/test_route_conflicts.py", + evidence=("pending_child_activity", "route_conflict_response"), + ), +) + +HIGH_VALUE_GAP_COVERAGE = ( + InteractionCoverage( + scenario_id="real_parent_action_fencing_unavailable_without_production_seam", + fields=("runtime_exclusivity", "workflow_filesystem_state", "websocket_api_contracts"), + invariants=("runtime.parent_action_fences_child_outputs",), + adapter="real_coordinator", + result="blocked", + test_file="tests/workflow_real_adapters/test_high_value_scenarios.py", + diagnostics={ + "reason": ( + "The invariant catalog declares model-only support and no existing isolated " + "production seam exposes stale child output acceptance after parent takeover." + ) + }, + ), + InteractionCoverage( + scenario_id="real_provider_stop_reset_checkpoint_unavailable_without_wait_seam", + fields=("provider_pause_resume", "workflow_filesystem_state", "runtime_exclusivity"), + invariants=("provider.stop_resume_preserves_pause",), + adapter="real_coordinator", + result="blocked", + test_file="tests/workflow_real_adapters/test_high_value_scenarios.py", + diagnostics={ + "reason": ( + "Existing coordinator coverage observes pause/reset resume, but no bounded " + "test seam independently interleaves user Stop while the provider wait is active." + ) + }, + ), +) + +HIGH_VALUE_REAL_COVERAGE = ( + InteractionCoverage( + scenario_id="real_disabled_proof_runtime_skips_autonomous_checkpoint", + fields=("proof_runtime_gating", "allowed_outputs", "workflow_filesystem_state"), + invariants=("proof_runtime.no_lean_when_disabled",), + adapter="real_coordinator", + result="passed", + test_file="tests/workflow_real_adapters/test_high_value_scenarios.py", + evidence=("disabled_runtime", "zero_proof_stage_calls"), + ), + InteractionCoverage( + scenario_id="real_hosted_proof_settings_returns_desktop_unavailable", + fields=("proof_runtime_gating", "websocket_api_contracts", "allowed_outputs"), + invariants=("proof_runtime.hosted_proof_settings_unavailable",), + adapter="real_route", + result="passed", + test_file="tests/workflow_real_adapters/test_high_value_scenarios.py", + evidence=("hosted_mode", "http_501"), + ), +) + +BUILD_BC_REAL_COVERAGE = ( + InteractionCoverage( + scenario_id="real_manual_aggregator_route_lifecycle_output_and_temp_root", + fields=("workflow_filesystem_state", "runtime_exclusivity", "prompt_context"), + invariants=("runtime.single_active_workflow",), + adapter="real_route", + result="passed", + test_file="tests/workflow_real_adapters/test_build_bc_lifecycles.py", + evidence=("route_start_stop", "saved_output", "temp_root_containment"), + ), + InteractionCoverage( + scenario_id="real_manual_aggregator_clear_archives_before_clear", + fields=("proof_scope_isolation", "workflow_filesystem_state"), + invariants=("proof_scope.manual_clear_archives_active",), + adapter="real_route", + result="passed", + test_file="tests/workflow_real_adapters/test_build_bc_lifecycles.py", + evidence=("archive_before_clear", "upload_cleanup"), + ), + InteractionCoverage( + scenario_id="real_actual_route_start_conflict_matrix_no_side_effects", + fields=("runtime_exclusivity", "workflow_filesystem_state"), + invariants=("runtime.single_active_workflow", "runtime.child_tasks_count_as_active"), + adapter="real_route", + result="passed", + test_file="tests/workflow_real_adapters/test_build_bc_lifecycles.py", + evidence=("sixteen_actual_route_calls", "pending_owner", "no_initialize_side_effects"), + ), + InteractionCoverage( + scenario_id="real_manual_aggregator_durable_hydration_and_clear_blocker", + fields=("workflow_filesystem_state", "proof_scope_isolation", "prompt_context"), + invariants=("state.runtime_roots_are_active_roots",), + adapter="real_coordinator", + result="passed", + test_file="tests/workflow_real_adapters/test_build_bc_lifecycles.py", + evidence=("persisted_result_hydration", "temp_root", "blocked_clear_no_archive"), + ), + InteractionCoverage( + scenario_id="real_manual_compiler_rejects_no_allowed_outputs", + fields=("allowed_outputs", "runtime_exclusivity"), + invariants=("outputs.at_least_one_output_enabled",), + adapter="real_route", + result="passed", + test_file="tests/workflow_real_adapters/test_build_bc_lifecycles.py", + evidence=("http_400", "coordinator_not_started"), + ), + InteractionCoverage( + scenario_id="real_generated_proof_appendix_stripping", + fields=("prompt_context", "proof_scope_isolation"), + invariants=("proof_scope.generated_appendices_stripped_from_prompts",), + adapter="real_coordinator", + result="passed", + test_file="tests/workflow_real_adapters/test_build_bc_lifecycles.py", + evidence=("brainstorm_appendix_stripped", "paper_appendix_stripped"), + ), + InteractionCoverage( + scenario_id="real_manual_compiler_papers_lifecycle_and_clear_archive", + fields=("allowed_outputs", "workflow_filesystem_state", "proof_scope_isolation"), + invariants=( + "proof_scope.manual_clear_archives_active", + "state.clear_removes_active_preserves_history", + ), + adapter="real_route", + result="passed", + test_file="tests/workflow_real_adapters/test_build_bc_lifecycles.py", + evidence=("papers_only_start_stop", "archive_before_appendix_and_paper_clear", "temp_archive_root"), + ), + InteractionCoverage( + scenario_id="real_manual_compiler_proof_only_background_ownership", + fields=("allowed_outputs", "runtime_exclusivity", "proof_runtime_gating"), + invariants=("runtime.child_tasks_count_as_active",), + adapter="real_route", + result="passed", + test_file="tests/workflow_real_adapters/test_build_bc_lifecycles.py", + evidence=("actual_proof_only_start_route", "live_background_task", "second_start_conflict"), + ), + InteractionCoverage( + scenario_id="real_leanoj_coordinator_skip_force_clear_and_intermediate_edit", + fields=("runtime_exclusivity", "workflow_filesystem_state", "proof_scope_isolation"), + invariants=("state.runtime_roots_are_active_roots",), + adapter="real_coordinator", + result="passed", + test_file="tests/workflow_real_adapters/test_build_bc_lifecycles.py", + evidence=("real_skip_consume", "real_force_consume", "state_preserved_then_cleared", "master_proof_temp_root"), + ), + InteractionCoverage( + scenario_id="real_shared_start_guard_representative_race", + fields=("runtime_exclusivity", "websocket_api_contracts"), + invariants=("runtime.single_active_workflow",), + adapter="real_route", + result="passed", + test_file="tests/workflow_real_adapters/test_build_bc_lifecycles.py", + evidence=("concurrent_start_race", "one_winner"), + ), + InteractionCoverage( + scenario_id="real_leanoj_full_final_loop_not_safely_bounded", + fields=("proof_runtime_gating", "workflow_filesystem_state", "websocket_api_contracts"), + invariants=("events.proof_verified_after_registration",), + adapter="real_coordinator", + result="blocked", + test_file="tests/workflow_real_adapters/test_build_bc_lifecycles.py", + diagnostics={ + "reason": ( + "The production final loop combines unbounded model iteration, Lean execution, " + "semantic review, persistence, and registration without a bounded route-level seam. " + "Direct registration ordering is covered elsewhere; publishing a full lifecycle pass " + "would require faking the transition owner rather than observing it." + ) + }, + ), +) + +BUILD_D_REAL_COVERAGE = ( + InteractionCoverage( + scenario_id="real_manual_aggregator_overflow_identity_persists_across_reload", + fields=("websocket_api_contracts", "workflow_filesystem_state", "provider_pause_resume"), + invariants=( + "events.context_overflow_route_identity", + "events.context_overflow_persists_across_reload", + ), + adapter="real_coordinator", + result="passed", + test_file="tests/workflow_real_adapters/test_build_d_overflow_lifecycles.py", + evidence=("configured_effective_route", "event_log_reload", "temp_root"), + ), + InteractionCoverage( + scenario_id="real_autonomous_overflow_terminal_stop_once", + fields=("websocket_api_contracts", "workflow_filesystem_state", "runtime_exclusivity"), + invariants=( + "events.context_overflow_route_identity", + "events.context_overflow_terminal_stop_once", + ), + adapter="real_coordinator", + result="passed", + test_file="tests/workflow_real_adapters/test_build_d_overflow_lifecycles.py", + evidence=("configured_effective_route", "single_terminal_stop"), + ), + InteractionCoverage( + scenario_id="real_proof_context_overflow_is_scoped_nonfatal", + fields=("websocket_api_contracts", "proof_runtime_gating", "provider_pause_resume"), + invariants=( + "events.context_overflow_route_identity", + "events.proof_context_overflow_nonfatal", + ), + adapter="real_coordinator", + result="passed", + test_file="tests/workflow_real_adapters/test_build_d_overflow_lifecycles.py", + evidence=("configured_effective_route", "fatal_false", "no_parent_stop"), + ), + InteractionCoverage( + scenario_id="real_proof_scope_matrix_isolated_under_temp_root", + fields=("proof_scope_isolation", "workflow_filesystem_state", "websocket_api_contracts"), + invariants=("proof_scope.manual_not_in_autonomous_current",), + adapter="real_route", + result="passed", + test_file="tests/workflow_real_adapters/test_build_d_proof_scope_filesystem.py", + evidence=( + "autonomous_current_and_history", + "manual_active_and_history", + "leanoj_current_and_history", + "pairwise_sentinels", + ), + ), + InteractionCoverage( + scenario_id="real_manual_archive_and_leanoj_clear_preserve_scope_contract", + fields=("proof_scope_isolation", "workflow_filesystem_state", "assistant_memory"), + invariants=( + "proof_scope.manual_clear_archives_active", + "state.clear_removes_active_preserves_history", + ), + adapter="real_coordinator", + result="passed", + test_file="tests/workflow_real_adapters/test_build_d_proof_scope_filesystem.py", + evidence=("physical_manual_archive", "active_store_empty", "leanoj_root_removed"), + ), + InteractionCoverage( + scenario_id="real_pruned_paper_preserved_but_removed_from_active_context", + fields=("workflow_filesystem_state", "prompt_context"), + invariants=("state.pruned_papers_excluded_from_context",), + adapter="real_route", + result="passed", + test_file="tests/workflow_real_adapters/test_build_d_proof_scope_filesystem.py", + evidence=( + "pruned_history_downloadable", + "active_enumeration_empty", + "brainstorm_reference_removed", + "rag_source_removed", + "temp_root", + ), + ), +) + +BUILD_E_REAL_COVERAGE = ( + InteractionCoverage( + scenario_id="real_direct_source_rag_exclusion_matrix", + fields=("prompt_context", "workflow_filesystem_state"), + invariants=("prompt.direct_sources_excluded_from_rag",), + adapter="real_coordinator", + result="passed", + test_file="tests/workflow_real_adapters/test_build_e_rag_prompt_context.py", + evidence=( + "aggregator_direct_and_offloaded_sources", + "compiler_construction_exclusions", + "compiler_rigor_exclusions", + "offloaded_source_retrievable", + ), + ), + InteractionCoverage( + scenario_id="real_model_visible_prompts_strip_generated_proof_appendices", + fields=("prompt_context", "proof_scope_isolation", "proof_runtime_gating"), + invariants=( + "prompt.generated_appendices_stripped", + "proof_scope.generated_appendices_stripped_from_prompts", + ), + adapter="real_coordinator", + result="passed", + test_file="tests/workflow_real_adapters/test_build_e_prompt_boundaries.py", + evidence=( + "paper_head_and_tail_preserved", + "generated_appendix_absent", + "verified_library_context_present", + "rigor_message_captured", + ), + ), + InteractionCoverage( + scenario_id="real_rigor_mandatory_source_overflow_fails_before_model_or_rag", + fields=("prompt_context", "proof_runtime_gating", "websocket_api_contracts"), + invariants=("prompt.mandatory_source_overflow_fails_visible",), + adapter="real_coordinator", + result="passed", + test_file="tests/workflow_real_adapters/test_build_e_prompt_boundaries.py", + evidence=("visible_value_error", "zero_model_calls", "zero_rag_calls", "source_unchanged"), + ), +) + +_UNLINKED_REAL_ADAPTER_COVERAGE = ( + ROUTE_CONFLICT_COVERAGE + + PROOF_ROUTE_COVERAGE + + HIGH_VALUE_REAL_COVERAGE + + BUILD_BC_REAL_COVERAGE + + BUILD_D_REAL_COVERAGE + + BUILD_E_REAL_COVERAGE + + HIGH_VALUE_GAP_COVERAGE +) + + +def _ranked_selectors_from_test_file(record: InteractionCoverage) -> tuple[str, ...]: + """Resolve exact pytest nodes from the uniquely matching scenario words.""" + import ast + from pathlib import Path + + test_path = Path(record.test_file) + source = test_path.read_text(encoding="utf-8") + functions = [ + node.name + for node in ast.parse(source).body + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) + and node.name.startswith("test_") + ] + scenario_words = set(record.scenario_id.removeprefix("real_").split("_")) + ranked = sorted( + functions, + key=lambda name: len(scenario_words.intersection(name.removeprefix("test_").split("_"))), + reverse=True, + ) + if not ranked: + raise AssertionError(f"No pytest selector found for {record.scenario_id}") + return tuple(f"{record.test_file}::{name}" for name in ranked) + + +def _link_passed_records() -> tuple[InteractionCoverage, ...]: + used_selectors: set[str] = set() + linked: list[InteractionCoverage] = [] + for record in _UNLINKED_REAL_ADAPTER_COVERAGE: + if record.result != "passed": + linked.append(record) + continue + selector = next( + ( + candidate + for candidate in _ranked_selectors_from_test_file(record) + if candidate not in used_selectors + ), + None, + ) + if selector is None: + raise AssertionError(f"No unique pytest selector found for {record.scenario_id}") + used_selectors.add(selector) + linked.append( + replace( + record, + runner="pytest", + test_selectors=(selector,), + asserted_invariants=record.invariants, + ) + ) + return tuple(linked) + + +REAL_ADAPTER_COVERAGE = _link_passed_records() diff --git a/tests/workflow_real_adapters/helpers.py b/tests/workflow_real_adapters/helpers.py new file mode 100644 index 0000000..07bf2dd --- /dev/null +++ b/tests/workflow_real_adapters/helpers.py @@ -0,0 +1,59 @@ +from __future__ import annotations + +from dataclasses import dataclass, field +from types import SimpleNamespace +from typing import Any, Callable + +from tests.workflow_harness.real_adapters import EventCollector + + +@dataclass +class CallRecorder: + calls: list[tuple[tuple[Any, ...], dict[str, Any]]] = field(default_factory=list) + result: Any = None + + def __call__(self, *args: Any, **kwargs: Any) -> Any: + self.calls.append((args, kwargs)) + return self.result + + async def async_call(self, *args: Any, **kwargs: Any) -> Any: + return self(*args, **kwargs) + + @property + def count(self) -> int: + return len(self.calls) + + +def inactive_workflow_flags() -> dict[str, object]: + return { + "coordinator": SimpleNamespace(is_running=False), + "compiler_coordinator": SimpleNamespace(is_running=False), + "autonomous_coordinator": SimpleNamespace( + is_active=False, + get_state=lambda: SimpleNamespace(is_running=False, current_tier="idle"), + ), + "leanoj_coordinator": SimpleNamespace(is_active=False), + } + + +def patch_attributes(monkeypatch, target: object, attributes: dict[str, object]) -> None: + for name, value in attributes.items(): + monkeypatch.setattr(target, name, value) + + +def assert_event_sequence( + collector: EventCollector, + *event_types: str, + predicate: Callable[[dict[str, Any]], bool] | None = None, +) -> None: + actual = [event_type for event_type, _payload in collector.events] + cursor = 0 + for expected in event_types: + try: + cursor = actual.index(expected, cursor) + 1 + except ValueError as exc: + raise AssertionError( + f"Expected ordered event {expected!r} after index {cursor}; observed {actual!r}." + ) from exc + if predicate is not None and not any(predicate(payload) for _, payload in collector.events): + raise AssertionError("No collected event payload satisfied the required predicate.") diff --git a/tests/workflow_real_adapters/test_build_bc_lifecycles.py b/tests/workflow_real_adapters/test_build_bc_lifecycles.py new file mode 100644 index 0000000..99c0c45 --- /dev/null +++ b/tests/workflow_real_adapters/test_build_bc_lifecycles.py @@ -0,0 +1,662 @@ +from __future__ import annotations + +from contextlib import asynccontextmanager +from types import SimpleNamespace +from unittest.mock import AsyncMock + +import pytest +from fastapi import HTTPException + +from backend.api.routes import aggregator as aggregator_route +from backend.api.routes import autonomous as autonomous_route +from backend.api.routes import compiler as compiler_route +from backend.api.routes import leanoj as leanoj_route +from backend.aggregator.core.coordinator import Coordinator +from backend.aggregator.memory.shared_training import SharedTrainingMemory +from backend.autonomous.memory.paper_library import PaperLibrary +from backend.autonomous.core.autonomous_coordinator import AutonomousCoordinator +from backend.compiler.core.compiler_coordinator import CompilerCoordinator +from backend.leanoj.core.leanoj_coordinator import LeanOJCoordinator +from backend.leanoj.core.leanoj_context import leanoj_context_manager +from backend.shared.config import system_config +from backend.shared.models import AutonomousResearchStartRequest, SubmitterConfig +from backend.shared.workflow_start_guard import WorkflowStartGuard +from tests.workflow_harness.real_adapters import ( + LeanOJAdapter, + ManualAggregatorAdapter, + ManualCompilerAdapter, + assert_single_race_winner, + minimal_aggregator_request, + minimal_compiler_request, + minimal_leanoj_request, + race_starts, +) +from tests.workflow_real_adapters.helpers import inactive_workflow_flags, patch_attributes + + +class _FakeSolutionPath: + def __init__(self) -> None: + self.stopped = 0 + + async def stop(self) -> None: + self.stopped += 1 + + +class _FakeManualAggregatorCoordinator: + def __init__(self) -> None: + self.is_running = False + self.solution_path_manager = _FakeSolutionPath() + self.initialized: dict = {} + self.cleared = 0 + + async def initialize(self, **kwargs) -> None: + self.initialized = kwargs + + async def start(self) -> None: + self.is_running = True + + async def stop(self) -> None: + self.is_running = False + + async def clear_all_submissions(self) -> None: + self.cleared += 1 + + async def get_results_formatted(self) -> str: + return "Submission #1\nA rigorous accepted result." + + +def _autonomous_request() -> AutonomousResearchStartRequest: + return AutonomousResearchStartRequest( + user_research_prompt="Research a rigorous solution.", + submitter_configs=[ + SubmitterConfig( + submitter_id=1, + model_id="test-submitter", + context_window=4096, + max_output_tokens=512, + ) + ], + allow_mathematical_proofs=False, + allow_research_papers=True, + validator_model="test-validator", + validator_context_window=4096, + validator_max_tokens=512, + writer_model="test-writer", + writer_context_window=4096, + writer_max_tokens=512, + high_param_model="test-rigor", + high_param_context_window=4096, + high_param_max_tokens=512, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("target", "active_mode"), + [ + (target, active) + for target in ("aggregator", "compiler", "autonomous", "leanoj") + for active in ("aggregator", "compiler", "autonomous", "leanoj") + ], +) +async def test_actual_route_start_conflict_matrix_has_no_start_side_effects( + monkeypatch, target, active_mode +): + aggregator = Coordinator() + compiler = CompilerCoordinator() + autonomous = AutonomousCoordinator() + leanoj = LeanOJCoordinator() + aggregator.is_running = active_mode == "aggregator" + compiler.is_running = active_mode == "compiler" + autonomous._state.is_running = active_mode == "autonomous" + autonomous._main_task = SimpleNamespace(done=lambda: active_mode != "autonomous") + leanoj._running = active_mode == "leanoj" + + route_modules = ( + aggregator_route, + compiler_route, + autonomous_route, + leanoj_route, + ) + for module in route_modules: + patch_attributes( + monkeypatch, + module, + { + "coordinator": aggregator, + "compiler_coordinator": compiler, + "autonomous_coordinator": autonomous, + "leanoj_coordinator": leanoj, + }, + ) + + forbidden = AsyncMock(side_effect=AssertionError("start side effect ran after conflict")) + monkeypatch.setattr(aggregator, "initialize", forbidden) + monkeypatch.setattr(compiler, "initialize", forbidden) + monkeypatch.setattr(autonomous, "initialize", forbidden) + monkeypatch.setattr(leanoj, "resume_or_initialize", forbidden) + + starts = { + "aggregator": lambda: aggregator_route.start_aggregator(minimal_aggregator_request()), + "compiler": lambda: compiler_route.start_compiler( + minimal_compiler_request( + allow_mathematical_proofs=False, + allow_research_papers=True, + ) + ), + "autonomous": lambda: autonomous_route.start_autonomous_research(_autonomous_request()), + "leanoj": lambda: leanoj_route.start_leanoj(minimal_leanoj_request()), + } + + with pytest.raises(HTTPException) as exc: + await starts[target]() + + assert exc.value.status_code == 400 + assert "running" in str(exc.value.detail).lower() + forbidden.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_actual_route_start_conflict_counts_pending_autonomous_ownership(monkeypatch): + aggregator = Coordinator() + compiler = CompilerCoordinator() + autonomous = AutonomousCoordinator() + leanoj = LeanOJCoordinator() + autonomous._state.is_running = False + autonomous._main_task = SimpleNamespace(done=lambda: False) + for module in (aggregator_route, compiler_route, leanoj_route): + patch_attributes( + monkeypatch, + module, + { + "coordinator": aggregator, + "compiler_coordinator": compiler, + "autonomous_coordinator": autonomous, + "leanoj_coordinator": leanoj, + }, + ) + forbidden = AsyncMock(side_effect=AssertionError("pending owner allowed start side effect")) + monkeypatch.setattr(aggregator, "initialize", forbidden) + monkeypatch.setattr(compiler, "initialize", forbidden) + monkeypatch.setattr(leanoj, "resume_or_initialize", forbidden) + + calls = ( + lambda: aggregator_route.start_aggregator(minimal_aggregator_request()), + lambda: compiler_route.start_compiler( + minimal_compiler_request( + allow_mathematical_proofs=False, + allow_research_papers=True, + ) + ), + lambda: leanoj_route.start_leanoj(minimal_leanoj_request()), + ) + for call in calls: + with pytest.raises(HTTPException) as exc: + await call() + assert exc.value.status_code == 400 + assert "Autonomous Research" in str(exc.value.detail) + forbidden.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_manual_aggregator_real_route_start_stop_output_and_temp_root(monkeypatch, tmp_path): + fake = _FakeManualAggregatorCoordinator() + flags = inactive_workflow_flags() + flags["coordinator"] = fake + patch_attributes(monkeypatch, aggregator_route, flags) + monkeypatch.setattr(system_config, "data_dir", tmp_path) + monkeypatch.setattr(system_config, "shared_training_file", tmp_path / "rag_shared_training.txt") + monkeypatch.setattr(aggregator_route, "save_manual_aggregator_prompt", AsyncMock()) + monkeypatch.setattr(aggregator_route, "require_embedding_provider_ready", AsyncMock()) + monkeypatch.setattr(aggregator_route, "_require_openrouter_host_provider_available", AsyncMock()) + monkeypatch.setattr(aggregator_route.api_client_manager, "configure_role", lambda *_args: None) + monkeypatch.setattr(aggregator_route.context_allocator, "set_context_windows", lambda *_args: None) + monkeypatch.setattr(aggregator_route.token_tracker, "reset", lambda: None) + monkeypatch.setattr(aggregator_route.token_tracker, "start_timer", lambda: None) + monkeypatch.setattr(aggregator_route.token_tracker, "stop_timer", lambda: None) + monkeypatch.setattr(aggregator_route.assistant_proof_search_coordinator, "stop_all", AsyncMock()) + + class _Registry: + async def acquire(self, *_args, **_kwargs): + return fake.solution_path_manager + + import backend.shared.solution_path as solution_path_module + + monkeypatch.setattr(solution_path_module, "solution_path_registry", _Registry()) + adapter = ManualAggregatorAdapter() + started = await adapter.start(minimal_aggregator_request()) + saved = await adapter.save_results() + stopped = await adapter.stop() + + assert started["status"] == "started" + assert fake.initialized["user_prompt"] == "Build a rigorous solution." + assert (tmp_path / saved["path"]).read_text(encoding="utf-8").startswith("Submission #1") + assert (tmp_path / saved["path"]).resolve().is_relative_to(tmp_path.resolve()) + assert stopped["status"] == "stopped" + assert fake.is_running is False + + +@pytest.mark.asyncio +async def test_manual_aggregator_clear_archives_before_destructive_clear(monkeypatch, tmp_path): + order: list[str] = [] + fake = _FakeManualAggregatorCoordinator() + fake.is_running = True + flags = inactive_workflow_flags() + flags["coordinator"] = fake + patch_attributes(monkeypatch, aggregator_route, flags) + monkeypatch.setattr(system_config, "data_dir", tmp_path) + + @asynccontextmanager + async def lock(): + yield + + async def archive(*_args, **_kwargs): + order.append("archive") + return 2 + + async def clear(): + order.append("clear") + fake.cleared += 1 + + fake.clear_all_submissions = clear + fake.solution_path_manager = None + monkeypatch.setattr(aggregator_route, "get_manual_proof_context_lock", lock) + monkeypatch.setattr(aggregator_route, "_manual_proof_clear_blocker", AsyncMock(return_value=None)) + monkeypatch.setattr(aggregator_route.manual_proof_database, "archive_current_run", archive) + monkeypatch.setattr(aggregator_route, "load_manual_aggregator_prompt", AsyncMock(return_value="prompt")) + monkeypatch.setattr(aggregator_route, "clear_manual_aggregator_prompt", AsyncMock()) + monkeypatch.setattr(aggregator_route, "_clear_uploaded_files", AsyncMock(return_value=3)) + monkeypatch.setattr(aggregator_route.assistant_proof_search_coordinator, "stop_all", AsyncMock()) + monkeypatch.setattr(aggregator_route.assistant_proof_search_coordinator, "clear_cooldown_state", AsyncMock()) + + result = await ManualAggregatorAdapter().clear() + + assert order == ["archive", "clear"] + assert result["archived_manual_proofs"] == 2 + assert result["deleted_uploads"] == 3 + + +@pytest.mark.asyncio +async def test_manual_aggregator_durable_results_hydrate_from_temp_root(monkeypatch, tmp_path): + persisted = tmp_path / "rag_shared_training.txt" + writer = SharedTrainingMemory() + writer.file_path = persisted + await writer.initialize() + await writer.add_accepted_submission("Durable accepted theorem route.") + + reader = SharedTrainingMemory() + reader.file_path = tmp_path / "stale-other-run.txt" + coordinator = Coordinator() + monkeypatch.setattr(system_config, "shared_training_file", persisted) + monkeypatch.setattr(aggregator_route, "shared_training_memory", reader) + monkeypatch.setattr(aggregator_route, "coordinator", coordinator) + monkeypatch.setattr( + coordinator, + "get_results_formatted", + lambda: reader.get_all_content_formatted(), + ) + monkeypatch.setattr( + aggregator_route, + "autonomous_coordinator", + SimpleNamespace(is_active=False), + ) + + result = await aggregator_route.get_results() + + assert "Durable accepted theorem route." in result["results"] + assert reader.file_path.resolve() == persisted.resolve() + assert reader.file_path.resolve().is_relative_to(tmp_path.resolve()) + + +@pytest.mark.asyncio +async def test_manual_aggregator_clear_blocker_has_no_destructive_side_effects(monkeypatch): + fake = _FakeManualAggregatorCoordinator() + archive = AsyncMock() + clear = AsyncMock() + fake.clear_all_submissions = clear + flags = inactive_workflow_flags() + flags["coordinator"] = fake + patch_attributes(monkeypatch, aggregator_route, flags) + + @asynccontextmanager + async def lock(): + yield + + monkeypatch.setattr(aggregator_route, "get_manual_proof_context_lock", lock) + monkeypatch.setattr( + aggregator_route, + "_manual_proof_clear_blocker", + AsyncMock(return_value="proof verification is active"), + ) + monkeypatch.setattr(aggregator_route.manual_proof_database, "archive_current_run", archive) + + with pytest.raises(HTTPException) as exc: + await aggregator_route.clear_all_submissions() + + assert exc.value.status_code == 409 + archive.assert_not_awaited() + clear.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_manual_compiler_rejects_invalid_output_selection_before_start(monkeypatch): + flags = inactive_workflow_flags() + patch_attributes(monkeypatch, compiler_route, flags) + request = minimal_compiler_request( + allow_mathematical_proofs=False, + allow_research_papers=False, + ) + + with pytest.raises(HTTPException) as exc: + await ManualCompilerAdapter().start(request) + + assert exc.value.status_code == 400 + assert exc.value.detail == "At least one allowed output must be enabled." + + +@pytest.mark.asyncio +async def test_manual_compiler_real_papers_only_route_start_stop(monkeypatch): + compiler = CompilerCoordinator() + aggregator = Coordinator() + autonomous = SimpleNamespace( + is_active=False, + get_state=lambda: SimpleNamespace(is_running=False), + ) + leanoj = SimpleNamespace(is_active=False) + patch_attributes( + monkeypatch, + compiler_route, + { + "compiler_coordinator": compiler, + "coordinator": aggregator, + "autonomous_coordinator": autonomous, + "leanoj_coordinator": leanoj, + }, + ) + initialized: dict = {} + + async def initialize(**kwargs): + initialized.update(kwargs) + + async def start(): + compiler.is_running = True + + async def stop(): + compiler.is_running = False + + monkeypatch.setattr(compiler, "initialize", initialize) + monkeypatch.setattr(compiler, "start", start) + monkeypatch.setattr(compiler, "stop", stop) + monkeypatch.setattr(compiler_route, "save_manual_compiler_prompt", AsyncMock()) + monkeypatch.setattr(compiler_route, "require_embedding_provider_ready", AsyncMock()) + monkeypatch.setattr(compiler_route.api_client_manager, "configure_role", lambda *_args, **_kwargs: None) + monkeypatch.setattr(compiler_route.token_tracker, "reset", lambda: None) + monkeypatch.setattr(compiler_route.token_tracker, "start_timer", lambda: None) + monkeypatch.setattr(compiler_route.token_tracker, "stop_timer", lambda: None) + monkeypatch.setattr(compiler_route.assistant_proof_search_coordinator, "stop_all", AsyncMock()) + + import backend.shared.solution_path as solution_path_module + + monkeypatch.setattr(solution_path_module.solution_path_registry, "get", lambda *_args: None) + adapter = ManualCompilerAdapter() + result = await adapter.start( + minimal_compiler_request( + allow_mathematical_proofs=False, + allow_research_papers=True, + ) + ) + stopped = await adapter.stop() + + assert result["status"] == "started" + assert initialized["allow_mathematical_proofs"] is False + assert initialized["compiler_prompt"] == "Write a rigorous paper." + assert stopped["status"] == "stopped" + assert compiler.is_running is False + + +@pytest.mark.asyncio +async def test_manual_compiler_proof_only_route_owns_background_task(monkeypatch): + import asyncio + + compiler = CompilerCoordinator() + aggregator = Coordinator() + autonomous = SimpleNamespace( + is_active=False, + get_state=lambda: SimpleNamespace(is_running=False), + ) + leanoj = SimpleNamespace(is_active=False) + patch_attributes( + monkeypatch, + compiler_route, + { + "compiler_coordinator": compiler, + "coordinator": aggregator, + "autonomous_coordinator": autonomous, + "leanoj_coordinator": leanoj, + }, + ) + monkeypatch.setattr(system_config, "generic_mode", False) + monkeypatch.setattr(system_config, "lean4_enabled", True) + entered = asyncio.Event() + release = asyncio.Event() + class Stage: + @classmethod + async def reserve_source(cls, source_type, source_id): + entered.set() + + async def bounded_check(_request, *, source_reserved=False): + assert source_reserved is True + await release.wait() + + monkeypatch.setattr(compiler_route, "ProofVerificationStage", Stage) + monkeypatch.setattr(compiler_route, "_run_compiler_aggregator_proof_check", bounded_check) + monkeypatch.setattr(compiler_route, "_compiler_proof_only_task", None) + + result = await compiler_route.start_compiler( + minimal_compiler_request( + allow_mathematical_proofs=True, + allow_research_papers=False, + ) + ) + + assert result["status"] == "proof_check_started" + assert entered.is_set() + task = compiler_route._compiler_proof_only_task + assert task is not None and not task.done() + assert "proof verification is already running" in compiler_route._get_start_conflict().lower() + release.set() + await task + assert task.done() + assert compiler_route.workflow_start_guard.active_owner is None + + +@pytest.mark.asyncio +async def test_manual_compiler_clear_archives_before_paper_clear(monkeypatch, tmp_path): + compiler = CompilerCoordinator() + compiler.user_prompt = "Durable compiler prompt" + compiler.is_running = True + order: list[str] = [] + monkeypatch.setattr(system_config, "data_dir", tmp_path) + monkeypatch.setattr(compiler_route, "compiler_coordinator", compiler) + + @asynccontextmanager + async def lock(): + yield + + async def stop(): + order.append("stop") + compiler.is_running = False + + async def archive(*args, **_kwargs): + order.append("archive") + assert args[0].resolve().is_relative_to(tmp_path.resolve()) + return 1 + + async def clear_appendix(): + order.append("appendix") + + async def clear_paper(): + order.append("paper") + + monkeypatch.setattr(compiler_route, "get_manual_proof_context_lock", lock) + monkeypatch.setattr(compiler_route, "_manual_proof_clear_blocker", AsyncMock(return_value=None)) + monkeypatch.setattr(compiler, "stop", stop) + monkeypatch.setattr(compiler, "clear_paper", clear_paper) + monkeypatch.setattr(compiler_route.manual_proof_database, "archive_current_run", archive) + monkeypatch.setattr(compiler_route, "clear_manual_shared_training_proof_appendix", clear_appendix) + monkeypatch.setattr(compiler_route, "clear_manual_compiler_prompt", AsyncMock()) + monkeypatch.setattr(compiler_route.assistant_proof_search_coordinator, "stop_all", AsyncMock()) + monkeypatch.setattr(compiler_route.assistant_proof_search_coordinator, "clear_cooldown_state", AsyncMock()) + import backend.shared.critique_memory as critique_module + + monkeypatch.setattr(critique_module, "clear_critiques", AsyncMock()) + + result = await ManualCompilerAdapter().clear() + + assert result["archived_manual_proofs"] == 1 + assert order == ["stop", "archive", "appendix", "paper"] + + +@pytest.mark.asyncio +async def test_leanoj_real_coordinator_skip_force_consume_and_route_clear(monkeypatch, tmp_path): + monkeypatch.setattr(system_config, "data_dir", tmp_path) + coordinator = LeanOJCoordinator() + coordinator.set_broadcast_callback(AsyncMock()) + await coordinator.initialize(minimal_leanoj_request()) + coordinator._state.phase = "recursive_brainstorm" + monkeypatch.setattr(leanoj_route, "leanoj_coordinator", coordinator) + monkeypatch.setattr(leanoj_route.assistant_proof_search_coordinator, "stop_all", AsyncMock()) + monkeypatch.setattr(leanoj_route.assistant_proof_search_coordinator, "clear_cooldown_state", AsyncMock()) + monkeypatch.setattr( + leanoj_context_manager, + "remove_all_leanoj_rag_sources", + AsyncMock(), + ) + adapter = LeanOJAdapter() + coordinator._running = True + assert (await adapter.skip_brainstorm())["success"] is True + assert await coordinator._consume_skip_brainstorm() is True + assert coordinator.get_state().phase == "final_proof_loop" + assert coordinator.get_state().user_forced_final_cycle is True + assert (await adapter.force_brainstorm())["success"] is True + assert await coordinator._consume_force_brainstorm() is True + assert coordinator.get_state().phase == "recursive_brainstorm" + assert coordinator.get_state().user_forced_final_cycle is False + coordinator._running = False + assert (await adapter.stop())["success"] is True + state_path = tmp_path / "leanoj_sessions" / coordinator.get_state().session_id / "state.json" + assert state_path.exists() + assert state_path.resolve().is_relative_to(tmp_path.resolve()) + assert (await adapter.clear())["success"] is True + assert not (tmp_path / "leanoj_sessions").exists() + assert coordinator.get_state().session_id == "" + + +@pytest.mark.asyncio +async def test_leanoj_real_intermediate_master_proof_edit_stays_in_temp_root(monkeypatch, tmp_path): + monkeypatch.setattr(system_config, "data_dir", tmp_path) + coordinator = LeanOJCoordinator() + coordinator.set_broadcast_callback(AsyncMock()) + await coordinator.initialize(minimal_leanoj_request()) + coordinator._state.phase = "final_proof_loop" + adapter = LeanOJAdapter(coordinator=coordinator) + proof = "import Mathlib\n\nexample : 1 = 1 := by\n rfl" + + await adapter.write_intermediate_master_proof(proof, summary="bounded intermediate edit") + + proof_path = tmp_path / "leanoj_sessions" / coordinator.get_state().session_id / "master_proof.lean" + assert proof_path.read_text(encoding="utf-8") == proof + assert proof_path.resolve().is_relative_to(tmp_path.resolve()) + assert coordinator.get_state().master_proof_version == 1 + assert coordinator.get_state().master_proof_last_edit_summary == "bounded intermediate edit" + + +def test_generated_proof_appendices_are_stripped_without_truncating_body(): + brainstorm = ( + "Submission #1\nEssential source result.\n\n" + "=== PROOFS GENERATED FROM THIS BRAINSTORM ===\nprivate generated proof" + ) + paper = ( + "Abstract\nVisible paper.\n\n" + "=== PROOFS ATTACHED TO THIS PAPER (Lean 4 Verified) ===\nprivate generated proof" + ) + assert compiler_route._strip_manual_aggregator_proof_appendix(brainstorm).strip().endswith( + "Essential source result." + ) + stripped_paper = PaperLibrary.strip_verified_proofs_from_content(paper) + assert "Visible paper." in stripped_paper + assert "private generated proof" not in stripped_paper + + +@pytest.mark.asyncio +async def test_actual_compiler_and_aggregator_route_start_race_has_one_winner(monkeypatch): + guard = WorkflowStartGuard() + monkeypatch.setattr(aggregator_route, "workflow_start_guard", guard) + monkeypatch.setattr(compiler_route, "workflow_start_guard", guard) + aggregator = Coordinator() + compiler = CompilerCoordinator() + autonomous = SimpleNamespace( + is_active=False, + get_state=lambda: SimpleNamespace(is_running=False), + ) + leanoj = SimpleNamespace(is_active=False) + for module in (aggregator_route, compiler_route): + monkeypatch.setattr(module, "coordinator", aggregator) + monkeypatch.setattr(module, "compiler_coordinator", compiler) + monkeypatch.setattr(module, "autonomous_coordinator", autonomous) + monkeypatch.setattr(module, "leanoj_coordinator", leanoj) + monkeypatch.setattr(aggregator_route, "save_manual_aggregator_prompt", AsyncMock()) + monkeypatch.setattr(compiler_route, "save_manual_compiler_prompt", AsyncMock()) + monkeypatch.setattr(aggregator_route, "require_embedding_provider_ready", AsyncMock()) + monkeypatch.setattr(compiler_route, "require_embedding_provider_ready", AsyncMock()) + monkeypatch.setattr(aggregator_route, "_require_openrouter_host_provider_available", AsyncMock()) + monkeypatch.setattr(aggregator_route.api_client_manager, "configure_role", lambda *_args, **_kwargs: None) + monkeypatch.setattr(compiler_route.api_client_manager, "configure_role", lambda *_args, **_kwargs: None) + monkeypatch.setattr(aggregator_route.context_allocator, "set_context_windows", lambda *_args: None) + monkeypatch.setattr(aggregator_route.token_tracker, "reset", lambda: None) + monkeypatch.setattr(aggregator_route.token_tracker, "start_timer", lambda: None) + monkeypatch.setattr(compiler_route.token_tracker, "reset", lambda: None) + monkeypatch.setattr(compiler_route.token_tracker, "start_timer", lambda: None) + + async def aggregator_initialize(**_kwargs): + return None + + async def aggregator_start(): + aggregator.is_running = True + + async def compiler_initialize(**_kwargs): + return None + + async def compiler_start(): + compiler.is_running = True + + monkeypatch.setattr(aggregator, "initialize", aggregator_initialize) + monkeypatch.setattr(aggregator, "start", aggregator_start) + monkeypatch.setattr(compiler, "initialize", compiler_initialize) + monkeypatch.setattr(compiler, "start", compiler_start) + + class Registry: + async def acquire(self, *_args, **_kwargs): + return None + + def get(self, *_args): + return None + + import backend.shared.solution_path as solution_path_module + + monkeypatch.setattr(solution_path_module, "solution_path_registry", Registry()) + + outcomes = await race_starts( + { + "aggregator": lambda: aggregator_route.start_aggregator(minimal_aggregator_request()), + "compiler": lambda: compiler_route.start_compiler( + minimal_compiler_request( + allow_mathematical_proofs=False, + allow_research_papers=True, + ) + ), + } + ) + winner = assert_single_race_winner(outcomes) + assert winner.name in {"aggregator", "compiler"} + assert sum((aggregator.is_running, compiler.is_running)) == 1 diff --git a/tests/workflow_real_adapters/test_build_d_overflow_lifecycles.py b/tests/workflow_real_adapters/test_build_d_overflow_lifecycles.py new file mode 100644 index 0000000..a4a2320 --- /dev/null +++ b/tests/workflow_real_adapters/test_build_d_overflow_lifecycles.py @@ -0,0 +1,218 @@ +from __future__ import annotations + +from unittest.mock import AsyncMock + +import pytest + +from backend.aggregator.core.context_allocator import ContextAllocationError +from backend.aggregator.core.coordinator import Coordinator +from backend.aggregator.memory.event_log import EventLog, event_log +from backend.autonomous.agents.proof_formalization_agent import ProofFormalizationAgent +from backend.autonomous.core.autonomous_coordinator import AutonomousCoordinator +from backend.autonomous.core.proof_verification_stage import ProofVerificationStage +from backend.autonomous.memory.research_metadata import research_metadata +from backend.shared.models import ModelConfig, ProofAttemptFeedback, ProofCandidate +from backend.shared.provider_errors import ProviderContextLengthError, ProviderRouteIdentity +from tests.workflow_harness.real_adapters import ( + EventCollector, + assert_event_count, + assert_fatal_context_overflow_event, + assert_no_events, + assert_proof_context_overflow_event, + assert_route_identity, +) + + +def _model_config(model_id: str) -> ModelConfig: + return ModelConfig( + model_id=model_id, + provider="openrouter", + context_window=8192, + max_output_tokens=1024, + ) + + +def _candidate() -> ProofCandidate: + return ProofCandidate(theorem_id="candidate-1", statement="True") + + +@pytest.mark.asyncio +async def test_manual_aggregator_fatal_overflow_preserves_route_and_reloads( + monkeypatch, + tmp_path, +): + coordinator = Coordinator() + coordinator.is_running = True + collector = EventCollector() + coordinator.websocket_broadcaster = collector.broadcast + event_log.file_path = tmp_path / "aggregator_event_log.txt" + event_log.events = [] + await event_log.initialize() + + route = ProviderRouteIdentity( + provider="lm_studio", + model="fallback-model:2", + role_id="aggregator_submitter_1", + task_id="agg_sub1_001", + host_provider="local-sibling", + route_kind="lm_studio_fallback", + configured_provider="openrouter", + configured_model="configured/model", + ) + provider_error = ProviderContextLengthError("provider context limit", route=route) + overflow = ContextAllocationError.from_provider_error( + provider_error, + "mandatory direct context overflow", + required_tokens=9000, + available_tokens=7000, + ) + monkeypatch.setattr( + "backend.aggregator.core.coordinator.api_client_manager.get_role_config", + lambda _role: _model_config("configured/model"), + ) + monkeypatch.setattr( + "backend.aggregator.core.coordinator.queue_manager.clear", + AsyncMock(), + ) + + await coordinator._handle_context_overflow( + overflow, + role_id="aggregator_submitter_1", + ) + + payload = assert_event_count( + collector.events, + "context_overflow_error", + 1, + )[0] + assert_fatal_context_overflow_event( + payload, + workflow_mode="aggregator", + role_id="aggregator_submitter_1", + ) + assert_route_identity( + payload, + configured_model="configured/model", + configured_provider="openrouter", + effective_model="fallback-model:2", + effective_provider="lm_studio", + effective_host_provider="local-sibling", + route_kind="lm_studio_fallback", + ) + assert coordinator.is_running is False + assert coordinator.fatal_error_payload == payload + assert event_log.file_path.resolve().is_relative_to(tmp_path.resolve()) + + reloaded = EventLog() + reloaded.file_path = event_log.file_path + await reloaded.initialize() + persisted = await reloaded.get_all_events() + assert len(persisted) == 1 + assert persisted[0]["type"] == "context_overflow_error" + assert persisted[0]["metadata"] == payload + + +@pytest.mark.asyncio +async def test_autonomous_fatal_overflow_terminal_stop_emits_exactly_once(monkeypatch): + coordinator = AutonomousCoordinator() + collector = EventCollector() + coordinator.set_broadcast_callback(collector.broadcast) + monkeypatch.setattr(research_metadata, "get_stats", AsyncMock(return_value={})) + payload = { + "workflow_mode": "autonomous", + "role_id": "compiler_writer", + "configured_model": "configured/model", + "configured_provider": "openrouter", + "effective_model": "fallback-model:2", + "effective_provider": "lm_studio", + "reason": "context_overflow", + "message": "Mandatory direct context exceeded the configured model window.", + "resolution": "Increase the configured context window or reduce mandatory context.", + } + + coordinator._mark_context_overflow_stop(payload) + await coordinator._broadcast_stopped_once() + await coordinator._broadcast_stopped_once() + + terminal = assert_event_count( + collector.events, + "auto_research_stopped", + 1, + )[0] + assert_fatal_context_overflow_event( + terminal, + workflow_mode="autonomous", + role_id="compiler_writer", + ) + assert_route_identity( + terminal, + configured_model="configured/model", + configured_provider="openrouter", + effective_model="fallback-model:2", + effective_provider="lm_studio", + ) + + +@pytest.mark.asyncio +async def test_nonfatal_proof_overflow_emits_once_without_parent_stop(monkeypatch): + feedback = ProofAttemptFeedback( + attempt=1, + theorem_id="candidate-1", + error_output="MANDATORY FULL SOURCE CONTEXT OVERFLOW", + configured_model="configured/model", + configured_provider="openrouter", + effective_model="fallback-model:2", + effective_provider="lm_studio", + overflow_origin="provider", + ) + + async def fake_prove(self, *args, attempt_callback=None, **kwargs): + await attempt_callback(feedback) + return False, "", "", [feedback] + + monkeypatch.setattr(ProofFormalizationAgent, "prove_candidate", fake_prove) + stage = ProofVerificationStage() + monkeypatch.setattr(stage, "_prepare_candidate", AsyncMock(return_value=_candidate())) + monkeypatch.setattr(stage, "_run_smt_check", AsyncMock(return_value=None)) + collector = EventCollector() + + await stage._run_lean_pipeline_for_candidate( + theorem_candidate=_candidate(), + base_event={"source_type": "paper", "source_id": "paper-1"}, + proof_label="A", + user_prompt="Prove the claim.", + source_type="paper", + source_id="paper-1", + source_content="A short source.", + source_title="Paper", + submitter_model="configured/model", + submitter_context=32_000, + submitter_max_tokens=2_000, + role_suffix="paper", + trigger="automatic", + novel_proofs_db=None, + broadcast_fn=collector.broadcast, + ) + + payload = assert_event_count( + collector.events, + "proof_context_overflow", + 1, + )[0] + assert_proof_context_overflow_event(payload, workflow_mode="autonomous") + assert_route_identity( + payload, + configured_model="configured/model", + configured_provider="openrouter", + effective_model="fallback-model:2", + effective_provider="lm_studio", + ) + assert_no_events( + collector.events, + "context_overflow_error", + "auto_research_stopped", + "compiler_stopped", + "aggregator_stopped", + "leanoj_stopped", + "proof_attempt_failed", + ) diff --git a/tests/workflow_real_adapters/test_build_d_proof_scope_filesystem.py b/tests/workflow_real_adapters/test_build_d_proof_scope_filesystem.py new file mode 100644 index 0000000..1526edf --- /dev/null +++ b/tests/workflow_real_adapters/test_build_d_proof_scope_filesystem.py @@ -0,0 +1,293 @@ +from __future__ import annotations + +import json +from datetime import datetime +from pathlib import Path +from unittest.mock import AsyncMock + +import pytest + +from backend.api.routes import autonomous as autonomous_route +from backend.api.routes import leanoj as leanoj_route +from backend.api.routes import proofs as proofs_route +from backend.autonomous.core.autonomous_rag_manager import autonomous_rag_manager +from backend.autonomous.memory.paper_library import PaperLibrary +from backend.autonomous.memory.proof_database import ProofDatabase +from backend.leanoj.core.leanoj_coordinator import LeanOJCoordinator +from backend.leanoj.core.leanoj_context import leanoj_context_manager +from backend.shared.config import system_config +from backend.shared.models import LeanOJSubproofRecord, PaperMetadata, ProofRecord +from tests.workflow_harness.real_adapters import minimal_leanoj_request +from tests.workflow_harness.real_adapters.filesystem_assertions import ( + assert_files_exist, + assert_paths_absent, + assert_paths_within, +) + + +def _proof(proof_id: str, statement: str, *, run_id: str) -> ProofRecord: + return ProofRecord( + proof_id=proof_id, + theorem_statement=statement, + theorem_name=f"{proof_id}_theorem", + source_type="brainstorm", + source_id=run_id, + source_title=run_id, + user_prompt=f"Prompt for {run_id}", + run_id=run_id, + lean_code=f"theorem {proof_id}_theorem : True := by trivial", + solver="Lean 4", + created_at=datetime(2026, 7, 13), + novel=True, + novelty_tier="mathematical_discovery", + novelty_reasoning="Isolated workflow-scope fixture.", + attempt_count=1, + ) + + +async def _database(base_dir: Path, *records: ProofRecord) -> ProofDatabase: + database = ProofDatabase() + database.set_base_dir(base_dir) + await database.initialize() + for record in records: + await database.add_proof(record) + return database + + +@pytest.mark.asyncio +async def test_autonomous_current_and_library_routes_do_not_read_manual_scope(monkeypatch, tmp_path): + data_root = tmp_path / "data" + sessions_root = data_root / "auto_sessions" + active_dir = sessions_root / "active-session" / "proofs" + history_dir = sessions_root / "history-session" / "proofs" + manual_dir = data_root / "manual_proofs" + autonomous = await _database( + active_dir, + _proof("proof_autonomous_current", "AUTONOMOUS CURRENT SENTINEL", run_id="active-session"), + ) + await _database( + history_dir, + _proof("proof_autonomous_history", "AUTONOMOUS HISTORY SENTINEL", run_id="history-session"), + ) + manual = await _database( + manual_dir, + _proof("proof_manual_private", "MANUAL PRIVATE SENTINEL", run_id="manual-run"), + ) + + monkeypatch.setattr(system_config, "data_dir", data_root) + monkeypatch.setattr(system_config, "auto_sessions_base_dir", sessions_root) + monkeypatch.setattr(proofs_route, "proof_database", autonomous) + monkeypatch.setattr(proofs_route, "manual_proof_database", manual) + + current = await proofs_route.list_proofs(scope="autonomous") + library = await proofs_route.get_proof_library(scope="autonomous") + current_statements = {item["theorem_statement"] for item in current["proofs"]} + library_statements = {item["theorem_statement"] for item in library["proofs"]} + + assert current_statements == {"AUTONOMOUS CURRENT SENTINEL"} + assert "AUTONOMOUS HISTORY SENTINEL" in library_statements + assert "MANUAL PRIVATE SENTINEL" not in current_statements | library_statements + assert current["scope"] == library["scope"] == "autonomous" + assert_paths_within(data_root, [active_dir, history_dir, manual_dir]) + + +@pytest.mark.asyncio +async def test_manual_clear_archive_moves_active_proofs_to_history_only(monkeypatch, tmp_path): + data_root = tmp_path / "data" + active_dir = data_root / "manual_proofs" + history_root = data_root / "manual_proof_runs" + manual = await _database( + active_dir, + _proof("proof_manual_active", "MANUAL ACTIVE SENTINEL", run_id="manual-active"), + ) + autonomous = await _database( + data_root / "autonomous-active", + _proof("proof_autonomous_private", "AUTONOMOUS PRIVATE SENTINEL", run_id="auto-active"), + ) + monkeypatch.setattr(system_config, "data_dir", data_root) + monkeypatch.setattr(proofs_route, "manual_proof_database", manual) + monkeypatch.setattr(proofs_route, "proof_database", autonomous) + + before = await proofs_route.list_proofs(scope="manual") + metadata = await manual.archive_current_run( + history_root, + user_prompt="Archived manual prompt.", + reason="build_d_clear", + ) + after = await proofs_route.list_proofs(scope="manual") + history = await proofs_route.get_proof_library(scope="manual") + + assert metadata is not None + assert [proof["theorem_statement"] for proof in before["proofs"]] == ["MANUAL ACTIVE SENTINEL"] + assert after["proofs"] == [] + assert {proof["theorem_statement"] for proof in history["proofs"]} == { + "MANUAL ACTIVE SENTINEL" + } + assert "AUTONOMOUS PRIVATE SENTINEL" not in json.dumps(history) + assert_files_exist( + data_root, + f"manual_proof_runs/{metadata['session_id']}/session_metadata.json", + f"manual_proof_runs/{metadata['session_id']}/proofs/proof_proof_manual_active.json", + "manual_proofs/proofs_index.json", + ) + assert_paths_absent(data_root, "manual_proofs/proof_proof_manual_active.json") + + +@pytest.mark.asyncio +async def test_leanoj_current_and_history_routes_are_disjoint_then_clear_removes_root( + monkeypatch, tmp_path +): + data_root = tmp_path / "data" + monkeypatch.setattr(system_config, "data_dir", data_root) + coordinator = LeanOJCoordinator() + coordinator.set_broadcast_callback(AsyncMock()) + await coordinator.initialize(minimal_leanoj_request()) + current_session = coordinator.get_state().session_id + coordinator.get_state().verified_subproofs.append( + LeanOJSubproofRecord( + subproof_id="current_subproof", + request="Current request", + theorem_or_lemma="LEANOJ CURRENT SENTINEL", + verified=True, + lean_code="theorem leanoj_current : True := by trivial", + ) + ) + await coordinator._persist_state() + + history_session = "history-session" + history_dir = data_root / "leanoj_sessions" / history_session + history_dir.mkdir(parents=True) + history_payload = { + "session_id": history_session, + "phase": "verified", + "updated_at": "2026-07-12T00:00:00", + "request": { + "user_prompt": "Historical LeanOJ prompt", + "lean_template": "example : True := by trivial", + }, + "verified_subproofs": [ + { + "subproof_id": "history_subproof", + "request": "History request", + "theorem_or_lemma": "LEANOJ HISTORY SENTINEL", + "verified": True, + "lean_code": "theorem leanoj_history : True := by trivial", + } + ], + } + (history_dir / "state.json").write_text(json.dumps(history_payload), encoding="utf-8") + monkeypatch.setattr(leanoj_route, "leanoj_coordinator", coordinator) + monkeypatch.setattr(leanoj_route.assistant_proof_search_coordinator, "stop_all", AsyncMock()) + monkeypatch.setattr( + leanoj_route.assistant_proof_search_coordinator, + "clear_cooldown_state", + AsyncMock(), + ) + monkeypatch.setattr( + leanoj_context_manager, + "remove_all_leanoj_rag_sources", + AsyncMock(), + ) + + current = await leanoj_route.get_leanoj_proofs() + history = await leanoj_route.get_leanoj_library() + + assert {proof["theorem_statement"] for proof in current["proofs"]} == { + "LEANOJ CURRENT SENTINEL" + } + assert {proof["theorem_statement"] for proof in history["proofs"]} == { + "LEANOJ HISTORY SENTINEL" + } + assert {session["session_id"] for session in history["sessions"]} == {history_session} + assert current_session not in {session["session_id"] for session in history["sessions"]} + assert_files_exist( + data_root, + f"leanoj_sessions/{current_session}/state.json", + f"leanoj_sessions/{history_session}/state.json", + ) + + cleared = await leanoj_route.clear_leanoj(confirm=True) + + assert cleared["success"] is True + assert_paths_absent(data_root, "leanoj_sessions", "leanoj_partial_proofs", "leanoj_artifacts") + + +@pytest.mark.asyncio +async def test_pruned_paper_moves_to_temp_history_and_route_excludes_active_file( + monkeypatch, tmp_path +): + data_root = tmp_path / "data" + papers_dir = data_root / "auto_sessions" / "session-prune" / "papers" + papers_dir.mkdir(parents=True) + library = PaperLibrary() + library._base_dir = papers_dir + library._archive_dir = papers_dir / "archive" + library._pruned_dir = papers_dir / "pruned" + library._archive_dir.mkdir() + library._pruned_dir.mkdir() + metadata = PaperMetadata( + paper_id="paper_scope", + title="Pruned Scope Paper", + status="complete", + source_brainstorm_ids=["topic-scope"], + ) + (papers_dir / "paper_paper_scope.txt").write_text("PRUNED PAPER SENTINEL", encoding="utf-8") + (papers_dir / "paper_paper_scope_metadata.json").write_text( + json.dumps(metadata.model_dump(mode="json"), default=str), + encoding="utf-8", + ) + monkeypatch.setattr(system_config, "data_dir", data_root) + monkeypatch.setattr(system_config, "auto_sessions_base_dir", data_root / "auto_sessions") + monkeypatch.setattr(system_config, "auto_papers_dir", data_root / "auto_papers") + monkeypatch.setattr(autonomous_route, "paper_library", library) + + class ScopedMetadata: + def __init__(self): + self.pruned: list[tuple[str, str, str]] = [] + + async def prune_paper(self, paper_id, *, reason, pruned_by): + self.pruned.append((paper_id, reason, pruned_by)) + + class ScopedBrainstorms: + def __init__(self): + self.removed: list[tuple[str, str]] = [] + + async def remove_paper_reference(self, topic_id, paper_id): + self.removed.append((topic_id, paper_id)) + + scoped_metadata = ScopedMetadata() + scoped_brainstorms = ScopedBrainstorms() + rag_removals = AsyncMock() + monkeypatch.setattr(autonomous_rag_manager, "remove_paper_from_rag", rag_removals) + + result = await autonomous_route._delete_autonomous_paper_from_scope( + session_id="session-prune", + scoped_paper_library=library, + scoped_brainstorm_memory=scoped_brainstorms, + scoped_research_metadata=scoped_metadata, + paper_id="paper_scope", + ) + response = await autonomous_route.get_pruned_paper_history() + + assert result["success"] is True + assert result["pruned"] is True + assert result["source_brainstorms"] == ["topic-scope"] + assert response["success"] is True + assert response["total_count"] == 1 + assert response["papers"][0]["paper_id"] == "paper_scope" + assert response["papers"][0]["status"] == "pruned" + assert await library.get_all_papers() == [] + assert await library.get_papers_summary() == [] + assert scoped_metadata.pruned[0][0] == "paper_scope" + assert scoped_brainstorms.removed == [("topic-scope", "paper_scope")] + rag_removals.assert_awaited_once_with("paper_scope") + assert_files_exist( + data_root, + "auto_sessions/session-prune/papers/pruned/pruned_paper_paper_scope.txt", + "auto_sessions/session-prune/papers/pruned/pruned_paper_paper_scope_metadata.json", + ) + assert_paths_absent( + data_root, + "auto_sessions/session-prune/papers/paper_paper_scope.txt", + "auto_sessions/session-prune/papers/paper_paper_scope_metadata.json", + ) diff --git a/tests/workflow_real_adapters/test_build_e_prompt_boundaries.py b/tests/workflow_real_adapters/test_build_e_prompt_boundaries.py new file mode 100644 index 0000000..bcea95e --- /dev/null +++ b/tests/workflow_real_adapters/test_build_e_prompt_boundaries.py @@ -0,0 +1,230 @@ +from __future__ import annotations + +from types import SimpleNamespace +from unittest.mock import AsyncMock + +import pytest + +from backend.autonomous.agents import proof_formalization_agent as formalization_module +from backend.autonomous.agents.proof_formalization_agent import ProofFormalizationAgent +from backend.autonomous.memory.paper_library import PaperLibrary +from backend.compiler.agents import high_param_submitter as high_param_module +from backend.compiler.agents.high_param_submitter import HighParamSubmitter +from backend.compiler.memory.paper_memory import ( + THEOREMS_APPENDIX_END, + THEOREMS_APPENDIX_START, +) +from backend.shared.models import ProofCandidate + + +class _ProofStore: + def __init__(self, proofs=()): + self._proofs = list(proofs) + + async def get_all_proofs(self): + return list(self._proofs) + + async def get_recent_failure_hints(self, *_args, **_kwargs): + return [] + + +def _completion(content: str) -> dict: + return {"choices": [{"message": {"content": content}, "finish_reason": "stop"}]} + + +@pytest.mark.asyncio +async def test_real_proof_formalization_message_keeps_source_boundaries_and_verified_library( + monkeypatch, +): + source_head = "ORDINARY SOURCE HEAD SENTINEL" + source_tail = "ORDINARY SOURCE TAIL SENTINEL" + generated_proof = "GENERATED PAPER PROOF MUST NOT BE MODEL VISIBLE" + raw_source = ( + f"{source_head}\n" + + ("ordinary mathematical source material\n" * 80) + + f"\n{THEOREMS_APPENDIX_START}\n" + + generated_proof + + f"\n{THEOREMS_APPENDIX_END}\n" + + source_tail + ) + # Real source readers apply this boundary before formalization. + source_for_model = PaperLibrary.strip_verified_proofs_from_content(raw_source) + + verified_library = "SEPARATELY INJECTED VERIFIED PROOF LIBRARY SENTINEL" + captured_messages = [] + + async def fake_completion(**kwargs): + captured_messages.extend(kwargs["messages"]) + return _completion( + '{"theorem_name":"boundary_test","lean_code":' + '"import Mathlib\\n\\ntheorem boundary_test : True := by trivial",' + '"reasoning":"test"}' + ) + + async def fake_check(_code, **_kwargs): + return SimpleNamespace(success=True, error_output="", goal_states="") + + monkeypatch.setattr( + formalization_module, + "_latest_assistant_pack_for_lean_attempts", + lambda: ("pack-hash", verified_library, []), + ) + monkeypatch.setattr( + formalization_module.api_client_manager, + "generate_completion", + fake_completion, + ) + monkeypatch.setattr( + formalization_module, + "get_lean4_client", + lambda: SimpleNamespace(check_proof=fake_check), + ) + monkeypatch.setattr( + formalization_module.assistant_proof_search_coordinator, + "mark_pack_consumed_by_solver", + lambda *_args, **_kwargs: None, + ) + + agent = ProofFormalizationAgent( + model_id="fake-model", + context_window=16_000, + max_output_tokens=1_000, + role_id="autonomous_proof_paper", + ) + success, _, _, _ = await agent.prove_candidate( + user_research_prompt="Prove the prompt.", + source_type="paper", + theorem_candidate=ProofCandidate( + theorem_id="boundary-test", + statement="True", + formal_sketch="By triviality.", + expected_novelty_tier="novel_variant", + prompt_relevance_rationale="Directly tests the requested claim.", + novelty_rationale="Boundary test.", + why_not_standard_known_result="Test fixture candidate.", + ), + source_content=source_for_model, + max_attempts=1, + source_title="Boundary Paper", + ) + + assert success is True + assert len(captured_messages) == 1 + model_visible = captured_messages[0]["content"] + assert source_head in model_visible + assert source_tail in model_visible + assert generated_proof not in model_visible + assert verified_library in model_visible + + +@pytest.mark.asyncio +async def test_real_compiler_rigor_message_strips_theorem_appendix_and_keeps_proof_summary( + monkeypatch, +): + paper_head = "ORDINARY PAPER HEAD SENTINEL" + paper_tail = "ORDINARY PAPER TAIL SENTINEL" + generated_theorem = "GENERATED THEOREM APPENDIX MUST NOT BE MODEL VISIBLE" + paper = ( + f"{paper_head}\n" + + ("ordinary paper body\n" * 40) + + f"{THEOREMS_APPENDIX_START}\n" + + generated_theorem + + f"\n{THEOREMS_APPENDIX_END}\n" + + f"{paper_tail}" + ) + verified_summary = "SEPARATELY INJECTED VERIFIED RIGOR PROOF SENTINEL" + proof_record = SimpleNamespace( + proof_id="proof-boundary", + novel=True, + theorem_statement=verified_summary, + ) + captured_messages = [] + + async def fake_completion(**kwargs): + captured_messages.extend(kwargs["messages"]) + return _completion('{"needs_theorem_work":false,"reasoning":"test"}') + + monkeypatch.setattr(high_param_module.outline_memory, "get_outline", AsyncMock(return_value="I. Body")) + monkeypatch.setattr(high_param_module.paper_memory, "get_paper", AsyncMock(return_value=paper)) + monkeypatch.setattr( + high_param_module.compiler_rag_manager, + "retrieve_for_mode", + AsyncMock(return_value=SimpleNamespace(text="")), + ) + monkeypatch.setattr( + high_param_module.api_client_manager, + "generate_completion", + fake_completion, + ) + monkeypatch.setattr( + high_param_module.lm_studio_client, + "cache_model_load_config", + AsyncMock(), + ) + monkeypatch.setattr(high_param_module.system_config, "compiler_high_param_context_window", 16_000) + monkeypatch.setattr(high_param_module.system_config, "compiler_high_param_max_output_tokens", 1_000) + + submitter = HighParamSubmitter( + "fake-model", + "Advance the target problem.", + validator_context_window=8_000, + validator_max_tokens=500, + proof_database_store=_ProofStore([proof_record]), + ) + submitter.context_window = 16_000 + submitter.max_output_tokens = 1_000 + submitter.available_input_tokens = 14_000 + submitter.set_source_material_context( + "SOURCE MATERIAL HEAD\n" + ("source body\n" * 30) + "SOURCE MATERIAL TAIL", + "Source brainstorm", + ) + + result = await submitter._step_discovery() + + assert result is None + assert len(captured_messages) == 1 + model_visible = captured_messages[0]["content"] + assert paper_head in model_visible + assert paper_tail in model_visible + assert "SOURCE MATERIAL HEAD" in model_visible + assert "SOURCE MATERIAL TAIL" in model_visible + assert generated_theorem not in model_visible + assert verified_summary in model_visible + + +@pytest.mark.asyncio +async def test_real_compiler_rigor_mandatory_full_source_overflow_is_visible_and_pre_llm( + monkeypatch, +): + source = "MANDATORY SOURCE HEAD\n" + ("large mandatory source token " * 5_000) + "\nMANDATORY SOURCE TAIL" + paper = "CURRENT PAPER HEAD\nordinary paper body\nCURRENT PAPER TAIL" + model_call = AsyncMock() + rag_call = AsyncMock() + + monkeypatch.setattr(high_param_module.outline_memory, "get_outline", AsyncMock(return_value="I. Body")) + monkeypatch.setattr(high_param_module.paper_memory, "get_paper", AsyncMock(return_value=paper)) + monkeypatch.setattr(high_param_module.api_client_manager, "generate_completion", model_call) + monkeypatch.setattr(high_param_module.compiler_rag_manager, "retrieve_for_mode", rag_call) + monkeypatch.setattr(high_param_module.system_config, "compiler_high_param_context_window", 2_000) + monkeypatch.setattr(high_param_module.system_config, "compiler_high_param_max_output_tokens", 200) + + submitter = HighParamSubmitter( + "fake-model", + "Advance the target problem.", + validator_context_window=2_000, + validator_max_tokens=200, + proof_database_store=_ProofStore(), + ) + submitter.context_window = 2_000 + submitter.max_output_tokens = 200 + submitter.available_input_tokens = 800 + submitter.set_source_material_context(source, "Mandatory source brainstorm") + + with pytest.raises(ValueError, match="mandatory full source context"): + await submitter._step_discovery() + + model_call.assert_not_awaited() + rag_call.assert_not_awaited() + assert submitter._source_material_context == source + assert submitter._source_material_context.startswith("MANDATORY SOURCE HEAD") + assert submitter._source_material_context.endswith("MANDATORY SOURCE TAIL") diff --git a/tests/workflow_real_adapters/test_build_e_rag_prompt_context.py b/tests/workflow_real_adapters/test_build_e_rag_prompt_context.py new file mode 100644 index 0000000..f1dda33 --- /dev/null +++ b/tests/workflow_real_adapters/test_build_e_rag_prompt_context.py @@ -0,0 +1,195 @@ +from __future__ import annotations + +from unittest.mock import AsyncMock + +import pytest + +from backend.aggregator.core.context_allocator import ContextAllocator +from backend.compiler.agents.high_param_submitter import HighParamSubmitter +from backend.compiler.agents.writer_submitter import WritingSubmitter +from backend.shared.config import system_config +from tests.workflow_harness.real_adapters.source_tagged_rag import SourceTaggedRagIndex + + +@pytest.mark.asyncio +async def test_build_e_aggregator_mixed_direct_and_offloaded_sources_are_excluded( + monkeypatch, +): + index = SourceTaggedRagIndex() + index.add("rag_shared_training.txt", "DUPLICATE SHARED") + index.add("upload.txt", "DUPLICATE UPLOAD") + index.add("local_training.txt", "OFFLOADED LOCAL EVIDENCE") + monkeypatch.setattr( + "backend.aggregator.core.context_allocator.rag_manager", + index, + ) + monkeypatch.setattr( + ContextAllocator, + "_get_shared_training_rag_sources", + lambda _self: ["rag_shared_training.txt"], + ) + monkeypatch.setattr( + "backend.aggregator.core.context_allocator.count_tokens", + lambda text: len(text.split()), + ) + + allocation = await ContextAllocator().allocate_submitter_context( + user_prompt="solve the target", + json_schema="{}", + system_prompt="return JSON", + shared_training_content="DIRECT SHARED", + local_training_content="local " * 100_000, + rejection_log_content="", + user_files_content={"upload.txt": "DIRECT UPLOAD"}, + chunk_size=256, + context_window=12_000, + max_output_tokens=512, + ) + + assert "DIRECT SHARED" in allocation["direct"] + assert "DIRECT UPLOAD" in allocation["direct"] + assert "OFFLOADED LOCAL EVIDENCE" in allocation["rag_context"].text + assert "DUPLICATE SHARED" not in allocation["rag_context"].text + assert "DUPLICATE UPLOAD" not in allocation["rag_context"].text + assert set(index.calls[-1]["exclude_sources"]) == { + "rag_shared_training.txt", + "upload.txt", + } + + +@pytest.mark.asyncio +async def test_build_e_aggregator_source_becomes_retrievable_when_offloaded( + monkeypatch, +): + index = SourceTaggedRagIndex() + index.add("rag_shared_training.txt", "SHARED SOURCE RETRIEVED") + monkeypatch.setattr( + "backend.aggregator.core.context_allocator.rag_manager", + index, + ) + monkeypatch.setattr( + ContextAllocator, + "_get_shared_training_rag_sources", + lambda _self: ["rag_shared_training.txt"], + ) + monkeypatch.setattr( + "backend.aggregator.core.context_allocator.count_tokens", + lambda text: len(text.split()), + ) + + allocation = await ContextAllocator().allocate_submitter_context( + user_prompt="solve the target", + json_schema="{}", + system_prompt="return JSON", + shared_training_content="shared " * 100_000, + local_training_content="", + rejection_log_content="", + user_files_content={}, + chunk_size=256, + context_window=12_000, + max_output_tokens=512, + ) + + assert "SHARED SOURCE RETRIEVED" in allocation["rag_context"].text + assert "rag_shared_training.txt" not in index.calls[-1]["exclude_sources"] + + +@pytest.mark.asyncio +async def test_build_e_compiler_construction_excludes_direct_sources_and_keeps_references( + monkeypatch, +): + index = SourceTaggedRagIndex() + index.add("compiler_outline.txt", "DUPLICATE OUTLINE") + index.add("compiler_paper.txt", "DUPLICATE PAPER") + index.add("brainstorm_topic.txt", "DUPLICATE BRAINSTORM") + index.add("reference_paper.txt", "ELIGIBLE REFERENCE EVIDENCE") + captured: dict[str, str] = {} + + monkeypatch.setattr( + "backend.compiler.agents.writer_submitter.compiler_rag_manager", + index, + ) + monkeypatch.setattr( + "backend.compiler.agents.writer_submitter.outline_memory.get_outline", + AsyncMock(return_value="DIRECT OUTLINE"), + ) + monkeypatch.setattr( + "backend.compiler.agents.writer_submitter.paper_memory.get_paper", + AsyncMock(return_value="DIRECT PAPER"), + ) + monkeypatch.setattr( + "backend.compiler.agents.writer_submitter.api_client_manager.prewarm_assistant_memory_context", + AsyncMock(), + ) + + async def generate_completion(**kwargs): + captured["prompt"] = kwargs["messages"][0]["content"] + return { + "choices": [ + { + "message": { + "content": ( + '{"needs_construction": false, "section_complete": false, ' + '"reasoning": "No edit."}' + ) + } + } + ] + } + + monkeypatch.setattr( + "backend.compiler.agents.writer_submitter.api_client_manager.generate_completion", + generate_completion, + ) + monkeypatch.setattr(system_config, "compiler_writer_context_window", 32_000) + monkeypatch.setattr(system_config, "compiler_writer_max_output_tokens", 1_000) + + submitter = WritingSubmitter("fake-model", "solve the target") + await submitter.submit_construction( + section_phase="body", + brainstorm_content="DIRECT BRAINSTORM", + brainstorm_source_name="brainstorm_topic.txt", + ) + + assert "DIRECT OUTLINE" in captured["prompt"] + assert "DIRECT PAPER" in captured["prompt"] + assert "DIRECT BRAINSTORM" in captured["prompt"] + assert "ELIGIBLE REFERENCE EVIDENCE" in captured["prompt"] + assert "DUPLICATE OUTLINE" not in captured["prompt"] + assert "DUPLICATE PAPER" not in captured["prompt"] + assert "DUPLICATE BRAINSTORM" not in captured["prompt"] + assert set(index.calls[-1]["exclude_sources"]) == { + "compiler_outline.txt", + "compiler_paper.txt", + "brainstorm_topic.txt", + } + + +@pytest.mark.asyncio +async def test_build_e_compiler_rigor_excludes_direct_outline_and_paper(monkeypatch): + index = SourceTaggedRagIndex() + index.add("compiler_outline.txt", "DUPLICATE RIGOR OUTLINE") + index.add("compiler_paper.txt", "DUPLICATE RIGOR PAPER") + index.add("reference_paper.txt", "RIGOR REFERENCE EVIDENCE") + monkeypatch.setattr( + "backend.compiler.agents.high_param_submitter.compiler_rag_manager", + index, + ) + monkeypatch.setattr(system_config, "compiler_high_param_context_window", 16_000) + monkeypatch.setattr(system_config, "compiler_high_param_max_output_tokens", 1_000) + + submitter = HighParamSubmitter("fake-model", "solve the target") + submitter.context_window = 16_000 + submitter.max_output_tokens = 1_000 + evidence = await submitter._build_rigor_rag_context( + query_seed="DIRECT OUTLINE DIRECT PAPER", + reserved_tokens=2_000, + ) + + assert "RIGOR REFERENCE EVIDENCE" in evidence + assert "DUPLICATE RIGOR OUTLINE" not in evidence + assert "DUPLICATE RIGOR PAPER" not in evidence + assert set(index.calls[-1]["exclude_sources"]) == { + "compiler_outline.txt", + "compiler_paper.txt", + } diff --git a/tests/workflow_real_adapters/test_build_f_deep_matrix.py b/tests/workflow_real_adapters/test_build_f_deep_matrix.py new file mode 100644 index 0000000..372dcc2 --- /dev/null +++ b/tests/workflow_real_adapters/test_build_f_deep_matrix.py @@ -0,0 +1,150 @@ +"""Environment-gated bounded repeated execution of mature real adapters.""" +from __future__ import annotations + +import os +from pathlib import Path + +import pytest + +from tests.workflow_harness.real_adapters.deep_runner import ( + minimal_deep_environment, + remove_variant_root, + run_deep_variant, + workspace_mutable_snapshot, +) +from tests.workflow_harness.real_adapters.maturity_registry import ( + AdapterMaturity, + descriptors_for, + validate_maturity_registry, +) + + +pytestmark = pytest.mark.skipif( + os.getenv("MOTO_REAL_ADAPTER_DEEP_TESTS") != "1", + reason="set MOTO_REAL_ADAPTER_DEEP_TESTS=1 to run the real-adapter deep matrix", +) + +WORKSPACE_ROOT = Path(__file__).resolve().parents[2] +DEEP_DESCRIPTORS = descriptors_for(AdapterMaturity.DEEP) +PURPOSEFUL_VARIANTS = ( + ("cold", "17"), + ("replay", "83"), +) +SENSITIVE_ENV_NAMES = { + "OPENROUTER_API_KEY", + "HTTP_PROXY", + "HTTPS_PROXY", + "ALL_PROXY", + "REQUESTS_CA_BUNDLE", + "SSL_CERT_FILE", + "AWS_ACCESS_KEY_ID", + "AWS_SECRET_ACCESS_KEY", +} + + +def test_build_f_maturity_registry_preserves_deep_and_blocked_cases() -> None: + validate_maturity_registry(workspace_root=WORKSPACE_ROOT) + blocked = descriptors_for(AdapterMaturity.BLOCKED) + assert len(DEEP_DESCRIPTORS) >= 6 + assert {descriptor.scenario_id for descriptor in blocked} == { + "real_parent_action_fencing_unavailable_without_production_seam", + "real_provider_stop_reset_checkpoint_unavailable_without_wait_seam", + "real_leanoj_full_final_loop_not_safely_bounded", + } + + +def _write_network_blocker(fake_root: Path) -> None: + fake_root.mkdir(parents=True) + (fake_root / "sitecustomize.py").write_text( + "import socket\n" + "def _blocked(*args, **kwargs):\n" + " raise AssertionError('Build F deep tests forbid external network access')\n" + "socket.create_connection = _blocked\n" + "_original_connect = socket.socket.connect\n" + "def _guarded_connect(self, address):\n" + " host = address[0] if isinstance(address, tuple) else str(address)\n" + " if host not in {'127.0.0.1', '::1', 'localhost'}:\n" + " raise AssertionError('Build F deep tests forbid external network access')\n" + " return _original_connect(self, address)\n" + "socket.socket.connect = _guarded_connect\n", + encoding="utf-8", + ) + + +def test_build_f_deep_environment_is_minimal_and_clears_sensitive_routes( + tmp_path: Path, +) -> None: + base = { + "PATH": os.environ.get("PATH", ""), + "SYSTEMROOT": os.environ.get("SYSTEMROOT", ""), + **{name: "must-not-pass-through" for name in SENSITIVE_ENV_NAMES}, + } + env = minimal_deep_environment( + base, + workspace_root=WORKSPACE_ROOT, + variant_root=tmp_path / "variant", + fake_root=tmp_path / "fake", + hash_seed="101", + ) + + assert SENSITIVE_ENV_NAMES.isdisjoint(env) + assert env["PYTHON_KEYRING_BACKEND"] == "keyring.backends.null.Keyring" + assert env["PYTHONHASHSEED"] == "101" + assert env["NO_PROXY"] == "127.0.0.1,localhost,::1" + + +@pytest.mark.parametrize( + "descriptor", + DEEP_DESCRIPTORS, + ids=lambda descriptor: descriptor.scenario_id, +) +def test_build_f_deep_real_adapter_matrix_is_bounded_repeatable_and_contained( + tmp_path: Path, + descriptor, +) -> None: + before_workspace = workspace_mutable_snapshot(WORKSPACE_ROOT) + observations = [] + variant_roots = [] + try: + for variant, hash_seed in PURPOSEFUL_VARIANTS: + variant_root = tmp_path / descriptor.scenario_id / variant + fake_root = variant_root / "external-blocker" + for root in ( + variant_root / "data", + variant_root / "logs", + variant_root / "tmp", + variant_root / "home", + ): + root.mkdir(parents=True) + _write_network_blocker(fake_root) + variant_roots.append(variant_root) + observations.append( + run_deep_variant( + descriptor, + workspace_root=WORKSPACE_ROOT, + variant_root=variant_root, + fake_root=fake_root, + variant=variant, + hash_seed=hash_seed, + ) + ) + + assert observations[0].hash_seed != observations[1].hash_seed + for observation in observations: + assert observation.returncode == 0, ( + f"{descriptor.scenario_id} ({observation.variant}) failed\n" + f"STDOUT:\n{observation.stdout}\nSTDERR:\n{observation.stderr}" + ) + assert not observation.workspace_changes, ( + f"{descriptor.scenario_id} ({observation.variant}) changed repository runtime state:\n" + + "\n".join(observation.workspace_changes) + ) + assert descriptor.repeat_contract is not None + assert observations[0].repeat_value( + descriptor.repeat_contract + ) == observations[1].repeat_value(descriptor.repeat_contract) + assert workspace_mutable_snapshot(WORKSPACE_ROOT) == before_workspace + finally: + for variant_root in variant_roots: + remove_variant_root(variant_root) + assert all(not root.exists() for root in variant_roots) diff --git a/tests/workflow_real_adapters/test_build_f_maturity_registry.py b/tests/workflow_real_adapters/test_build_f_maturity_registry.py new file mode 100644 index 0000000..2ef1e85 --- /dev/null +++ b/tests/workflow_real_adapters/test_build_f_maturity_registry.py @@ -0,0 +1,134 @@ +from __future__ import annotations + +from pathlib import Path +import subprocess +import sys + +from tests.workflow_harness.real_adapters.maturity_registry import ( + AdapterMaturity, + REAL_ADAPTER_MATURITY_REGISTRY, + RepeatContract, + descriptors_for, + validate_maturity_registry, +) +from tests.workflow_real_adapters.coverage_records import REAL_ADAPTER_COVERAGE + + +WORKSPACE_ROOT = Path(__file__).resolve().parents[2] +EXPECTED_DEEP_IDS = { + "real_actual_route_start_conflict_matrix_no_side_effects", + "real_shared_start_guard_representative_race", + "real_proof_scope_matrix_isolated_under_temp_root", + "real_direct_source_rag_exclusion_matrix", + "real_model_visible_prompts_strip_generated_proof_appendices", + "real_rigor_mandatory_source_overflow_fails_before_model_or_rag", +} +EXPECTED_BLOCKED_IDS = { + "real_parent_action_fencing_unavailable_without_production_seam", + "real_provider_stop_reset_checkpoint_unavailable_without_wait_seam", + "real_leanoj_full_final_loop_not_safely_bounded", +} + + +def test_build_f_registry_describes_every_build_b_through_e_real_scenario(): + validate_maturity_registry(workspace_root=WORKSPACE_ROOT) + + assert {item.scenario_id for item in REAL_ADAPTER_MATURITY_REGISTRY} == { + item.scenario_id for item in REAL_ADAPTER_COVERAGE + } + assert descriptors_for(AdapterMaturity.NORMAL) + assert descriptors_for(AdapterMaturity.DEEP) + + +def test_build_f_maturity_matches_coverage_result_and_exact_partitions(): + blocked = descriptors_for(AdapterMaturity.BLOCKED) + deep = descriptors_for(AdapterMaturity.DEEP) + normal = descriptors_for(AdapterMaturity.NORMAL) + coverage_by_id = {item.scenario_id: item for item in REAL_ADAPTER_COVERAGE} + + assert {item.scenario_id for item in blocked} == EXPECTED_BLOCKED_IDS + assert {item.scenario_id for item in deep} == EXPECTED_DEEP_IDS + partitions = [ + {item.scenario_id for item in maturity} + for maturity in (normal, deep, blocked) + ] + assert all( + left.isdisjoint(right) + for index, left in enumerate(partitions) + for right in partitions[index + 1 :] + ) + assert set().union(*partitions) == set(coverage_by_id) + assert all(coverage_by_id[item.scenario_id].result == "blocked" for item in blocked) + assert all( + coverage_by_id[item.scenario_id].result == "passed" + for item in (*normal, *deep) + ) + assert all(not item.executable for item in blocked) + assert all(not item.repeat_safe for item in blocked) + assert all(item.blocked_reason for item in blocked) + + +def test_build_f_deep_promotions_are_repeat_safe_bounded_and_cleanup_guarded(): + deep = descriptors_for(AdapterMaturity.DEEP) + + assert len(deep) >= 6 + for descriptor in deep: + assert descriptor.executable + assert descriptor.repeat_safe + assert 0 < descriptor.timeout_seconds <= 180 + assert descriptor.cleanup.uses_temporary_roots + assert descriptor.cleanup.removes_runtime_state + assert descriptor.cleanup.forbids_workspace_escape + assert descriptor.pytest_args()[0] == "-q" + assert descriptor.repeat_contract in { + RepeatContract.NORMALIZED_PROCESS_RESULT, + RepeatContract.NORMALIZED_PROCESS_AND_RUNTIME_STATE, + } + assert descriptor.repeat_contract_reason + + +def test_build_f_all_executable_selectors_are_exact_collectable_node_ids(tmp_path): + executable = [ + descriptor + for descriptor in REAL_ADAPTER_MATURITY_REGISTRY + if descriptor.executable + ] + selectors = [ + selector + for descriptor in executable + for selector in descriptor.test_selectors + ] + + assert selectors + assert len(selectors) == len(set(selectors)) + assert all(selector.count("::") == 1 for selector in selectors) + completed = subprocess.run( + [ + sys.executable, + "-m", + "pytest", + "--collect-only", + "-q", + "-p", + "no:cacheprovider", + "--basetemp", + str(tmp_path / "collect"), + *selectors, + ], + cwd=WORKSPACE_ROOT, + capture_output=True, + text=True, + timeout=120, + check=False, + ) + + assert completed.returncode == 0, ( + f"selector collection failed\nSTDOUT:\n{completed.stdout}\n" + f"STDERR:\n{completed.stderr}" + ) + collected = { + line.strip().replace("\\", "/").split("[", 1)[0] + for line in completed.stdout.splitlines() + if "::test_" in line + } + assert collected == set(selectors) diff --git a/tests/workflow_real_adapters/test_coverage_metadata.py b/tests/workflow_real_adapters/test_coverage_metadata.py new file mode 100644 index 0000000..78e57ed --- /dev/null +++ b/tests/workflow_real_adapters/test_coverage_metadata.py @@ -0,0 +1,62 @@ +from __future__ import annotations + +from tests.workflow_harness.coverage_metadata import assert_coverage_records_valid +from tests.workflow_harness.invariant_catalog import INVARIANTS_BY_ID, invariant_ids_for_adapter +from tests.workflow_real_adapters.coverage_records import REAL_ADAPTER_COVERAGE + + +def test_real_adapter_coverage_metadata_is_valid(): + assert_coverage_records_valid(REAL_ADAPTER_COVERAGE) + + +def test_passed_real_coverage_links_exact_pytest_nodes_and_claimed_invariants(): + for record in REAL_ADAPTER_COVERAGE: + if record.result == "blocked": + assert record.runner is None + assert record.test_selectors == () + assert record.asserted_invariants == () + continue + assert record.runner == "pytest" + assert record.test_selectors + assert set(record.asserted_invariants) == set(record.invariants) + + +def test_first_build_real_adapter_coverage_advertises_landed_scenarios(): + covered_by_real_route = { + invariant_id + for record in REAL_ADAPTER_COVERAGE + if record.adapter == "real_route" and record.result != "blocked" + for invariant_id in record.invariants + } + covered_by_real_coordinator = { + invariant_id + for record in REAL_ADAPTER_COVERAGE + if record.adapter == "real_coordinator" and record.result != "blocked" + for invariant_id in record.invariants + } + + assert "runtime.single_active_workflow" in covered_by_real_route + assert { + "outputs.proofs_only_no_paper_phase", + "provider.pause_preserves_checkpoint", + "proof_scope.manual_not_in_autonomous_current", + }.issubset(covered_by_real_coordinator) + assert covered_by_real_route.issubset(invariant_ids_for_adapter("real_route")) + assert covered_by_real_coordinator.issubset(invariant_ids_for_adapter("real_coordinator")) + + +def test_plan2_real_adapter_gaps_are_explicitly_model_only_or_future_work(): + real_passed = { + invariant_id + for record in REAL_ADAPTER_COVERAGE + if record.result == "passed" + for invariant_id in record.invariants + } + model_only_or_future = set(INVARIANTS_BY_ID) - real_passed + + assert model_only_or_future + assert { + "runtime.parent_action_fences_child_outputs", + "proof_runtime.no_smt_when_disabled", + "prompt.proof_source_context_required", + }.issubset(model_only_or_future) diff --git a/tests/workflow_real_adapters/test_helpers.py b/tests/workflow_real_adapters/test_helpers.py new file mode 100644 index 0000000..412945f --- /dev/null +++ b/tests/workflow_real_adapters/test_helpers.py @@ -0,0 +1,47 @@ +from __future__ import annotations + +import pytest + +from tests.workflow_harness.real_adapters import EventCollector +from tests.workflow_real_adapters.helpers import CallRecorder, assert_event_sequence + + +@pytest.mark.asyncio +async def test_call_recorder_captures_sync_and_async_adapter_calls(): + recorder = CallRecorder(result={"accepted": True}) + + sync_result = recorder("source", scope="manual") + async_result = await recorder.async_call("checkpoint", resumed=True) + + assert sync_result == {"accepted": True} + assert async_result == {"accepted": True} + assert recorder.count == 2 + assert recorder.calls == [ + (("source",), {"scope": "manual"}), + (("checkpoint",), {"resumed": True}), + ] + + +@pytest.mark.asyncio +async def test_event_sequence_accepts_intervening_events_and_checks_payload(): + collector = EventCollector() + await collector.broadcast("provider_paused", {"reason": "credit", "scope": "autonomous"}) + await collector.broadcast("checkpoint_saved", {"source_id": "topic-1"}) + await collector.broadcast("provider_resumed", {"reason": "reset", "scope": "autonomous"}) + + assert_event_sequence( + collector, + "provider_paused", + "provider_resumed", + predicate=lambda payload: payload.get("source_id") == "topic-1", + ) + + +@pytest.mark.asyncio +async def test_event_sequence_rejects_reversed_provider_lifecycle(): + collector = EventCollector() + await collector.broadcast("provider_resumed", {"reason": "reset"}) + await collector.broadcast("provider_paused", {"reason": "credit"}) + + with pytest.raises(AssertionError, match="Expected ordered event"): + assert_event_sequence(collector, "provider_paused", "provider_resumed") diff --git a/tests/workflow_real_adapters/test_high_value_scenarios.py b/tests/workflow_real_adapters/test_high_value_scenarios.py new file mode 100644 index 0000000..2cca417 --- /dev/null +++ b/tests/workflow_real_adapters/test_high_value_scenarios.py @@ -0,0 +1,53 @@ +from __future__ import annotations + +from unittest.mock import AsyncMock + +import pytest +from fastapi import HTTPException + +from backend.api.routes import proofs as proofs_route +from backend.autonomous.core.autonomous_coordinator import AutonomousCoordinator +from backend.shared.config import system_config +from backend.shared.models import ProofSettingsUpdateRequest +from tests.workflow_real_adapters.coverage_records import ( + HIGH_VALUE_GAP_COVERAGE, + HIGH_VALUE_REAL_COVERAGE, +) + + +@pytest.mark.asyncio +async def test_disabled_proof_runtime_skips_autonomous_checkpoint(monkeypatch): + coordinator = AutonomousCoordinator() + stage = AsyncMock() + coordinator._proof_verification_stage = stage + coordinator._allow_mathematical_proofs = True + monkeypatch.setattr(system_config, "lean4_enabled", False) + + result = await coordinator._run_proof_verification( + "Proof-bearing source.", + "brainstorm", + "disabled-runtime-source", + source_title="Disabled runtime", + ) + + assert result == "complete" + stage.run.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_hosted_proof_settings_returns_desktop_unavailable(monkeypatch): + monkeypatch.setattr(system_config, "generic_mode", True) + request = ProofSettingsUpdateRequest(enabled=True, timeout=600) + + with pytest.raises(HTTPException) as raised: + await proofs_route.update_proof_settings(request) + + assert raised.value.status_code == 501 + assert raised.value.detail["lean4_enabled"] is False + assert "unavailable in hosted mode" in raised.value.detail["message"] + + +def test_unavailable_real_seams_are_truthfully_blocked(): + assert {record.result for record in HIGH_VALUE_GAP_COVERAGE} == {"blocked"} + assert all(record.diagnostics and record.diagnostics.get("reason") for record in HIGH_VALUE_GAP_COVERAGE) + assert {record.result for record in HIGH_VALUE_REAL_COVERAGE} == {"passed"} diff --git a/tests/workflow_real_adapters/test_proof_routes.py b/tests/workflow_real_adapters/test_proof_routes.py new file mode 100644 index 0000000..9208d9e --- /dev/null +++ b/tests/workflow_real_adapters/test_proof_routes.py @@ -0,0 +1,689 @@ +from __future__ import annotations + +import asyncio +from importlib import import_module +from unittest.mock import AsyncMock + +import pytest + +from backend.api.routes import compiler as compiler_route +from backend.autonomous.core.autonomous_coordinator import AutonomousCoordinator +from backend.autonomous.memory.research_metadata import ResearchMetadata +from backend.leanoj.core.leanoj_coordinator import LeanOJCoordinator +from backend.shared.config import system_config +from backend.shared.models import CompilerStartRequest, LeanOJRoleConfig, LeanOJStartRequest +from backend.shared.models import ProofCandidate +from backend.shared.provider_pause import resume_provider_pauses +from tests.workflow_real_adapters.coverage_records import PROOF_ROUTE_COVERAGE +from tests.workflow_real_adapters.helpers import CallRecorder, assert_event_sequence +from tests.workflow_harness.invariants import ( + assert_observed_invariant, + assert_proofs_only_never_enters_paper_phase, +) +from tests.workflow_harness.model import WorkflowMode, WorkflowPhase +from tests.workflow_harness.real_adapters import ( + EventCollector, + FakeResearchMetadata, + FakeProofStage, + RealWorkflowObservation, + RoleConfigCapture, +) + +coordinator_module = import_module("backend.autonomous.core.autonomous_coordinator") + + +def _compiler_request(**overrides) -> CompilerStartRequest: + data = { + "compiler_prompt": "Prove something from the manual Aggregator database.", + "allow_mathematical_proofs": True, + "allow_research_papers": False, + "validator_model": "validator-model", + "validator_context_size": 1000, + "validator_max_output_tokens": 100, + "writer_model": "writer-model", + "writer_context_size": 1000, + "writer_max_output_tokens": 100, + "high_param_provider": "openrouter", + "high_param_model": "rigor-model", + "high_param_openrouter_provider": "RigorHost", + "high_param_openrouter_reasoning_effort": "high", + "high_param_lm_studio_fallback": "rigor-fallback", + "high_param_context_size": 3000, + "high_param_max_output_tokens": 300, + "high_param_supercharge_enabled": True, + } + data.update(overrides) + return CompilerStartRequest(**data) + + +def _leanoj_request() -> LeanOJStartRequest: + role = LeanOJRoleConfig( + model_id="leanoj-test-model", + context_window=8192, + max_output_tokens=1024, + ) + return LeanOJStartRequest( + user_prompt="Prove one equals one.", + lean_template="import Mathlib\n\nexample : 1 = 1 := by\n sorry", + topic_generator=role, + topic_validator=role, + brainstorm_submitters=[role], + brainstorm_validator=role, + path_decider=role, + final_solver=role, + ) + + +@pytest.mark.asyncio +async def test_manual_compiler_proof_only_uses_rigor_settings_and_manual_scope(monkeypatch, tmp_path): + events = EventCollector() + roles = RoleConfigCapture() + fake_stage = FakeProofStage() + observation = RealWorkflowObservation( + runtime_root=tmp_path, + mode=WorkflowMode.MANUAL_COMPILER, + phase=WorkflowPhase.MANUAL_PROOF, + allow_mathematical_proofs=True, + allow_research_papers=False, + ) + request = _compiler_request() + + class StageFactory: + def __call__(self): + return fake_stage + + class ManualProofDb: + def inject_into_prompt(self, prompt: str) -> str: + return f"manual-scope::{prompt}" + + async def append_manual_proof(_proof): + observation.ingest_event( + "proof_verified", + {"proof_id": "manual-proof-1", "scope": "manual", "phase": WorkflowPhase.MANUAL_PROOF.value}, + ) + + monkeypatch.setattr( + compiler_route, + "_read_manual_aggregator_context", + AsyncMock(return_value="Accepted manual Aggregator submission."), + raising=False, + ) + monkeypatch.setattr(compiler_route, "_release_pre_reserved_source", AsyncMock(), raising=False) + monkeypatch.setattr(compiler_route.websocket, "broadcast_event", events.broadcast) + monkeypatch.setattr(compiler_route.api_client_manager, "configure_role", roles.configure) + monkeypatch.setattr(compiler_route, "ProofVerificationStage", StageFactory()) + monkeypatch.setattr(compiler_route, "manual_proof_database", ManualProofDb()) + monkeypatch.setattr(compiler_route, "append_proof_to_manual_shared_training", append_manual_proof) + monkeypatch.setattr(compiler_route.assistant_proof_search_coordinator, "stop_all", AsyncMock()) + monkeypatch.setattr(compiler_route.token_tracker, "reset", lambda: None) + monkeypatch.setattr(compiler_route.token_tracker, "start_timer", lambda: None) + monkeypatch.setattr(compiler_route.token_tracker, "stop_timer", lambda: None) + + await compiler_route._run_compiler_aggregator_proof_check(request, source_reserved=True) + + proof_role = roles.roles["autonomous_proof_formalization_compiler_aggregator"] + assert proof_role.provider == "openrouter" + assert proof_role.model_id == "rigor-model" + assert proof_role.openrouter_provider == "RigorHost" + assert proof_role.openrouter_reasoning_effort == "high" + assert proof_role.lm_studio_fallback_id == "rigor-fallback" + assert proof_role.context_window == 3000 + assert proof_role.max_output_tokens == 300 + assert proof_role.supercharge_enabled is True + assert fake_stage.calls[0]["submitter_model"] == "rigor-model" + assert fake_stage.calls[0]["novel_proofs_db"].inject_into_prompt("x") == "manual-scope::x" + observation.record("manual_compiler_proof_only") + assert observation.manual_proofs_active == {"manual-proof-1"} + observation.observed_invariants.add("proof_scope.manual_not_in_autonomous_current") + assert_observed_invariant(observation, "proof_scope.manual_not_in_autonomous_current") + + +@pytest.mark.asyncio +async def test_autonomous_proofs_only_brainstorm_handoff_returns_to_topic_exploration(monkeypatch, tmp_path): + events = EventCollector() + coordinator = AutonomousCoordinator() + coordinator._allow_mathematical_proofs = True + coordinator._allow_research_papers = False + coordinator._current_topic_id = "topic-proof-only" + coordinator._current_paper_id = "stale-paper" + coordinator._current_paper_title = "Stale paper" + coordinator._current_reference_papers = ["paper-ref"] + coordinator._current_reference_brainstorms = ["brainstorm-ref"] + coordinator._state.current_tier = "tier2_paper_writing" + coordinator._broadcast = events.broadcast + + saved_states = [] + + async def save_state(state): + saved_states.append(dict(state)) + + monkeypatch.setattr(coordinator_module.research_metadata, "set_current_brainstorm", AsyncMock()) + monkeypatch.setattr(coordinator_module.research_metadata, "save_workflow_state", save_state) + monkeypatch.setattr(coordinator_module.research_metadata, "get_workflow_state", AsyncMock(return_value={})) + monkeypatch.setattr( + coordinator_module.final_answer_memory, + "get_state", + lambda: type("Tier3State", (), {"is_active": False})(), + ) + monkeypatch.setattr(coordinator_module.final_answer_memory, "get_answer_format", lambda: None) + + await coordinator._handle_papers_disabled_after_brainstorm() + + observation = RealWorkflowObservation( + runtime_root=tmp_path, + mode=WorkflowMode.AUTONOMOUS, + phase=WorkflowPhase.TOPIC_EXPLORATION, + allow_mathematical_proofs=True, + allow_research_papers=False, + ) + observation.record("autonomous_proofs_only_handoff") + for event_type, payload in events.events: + observation.ingest_event(event_type, payload) + observation.checkpoint = saved_states[-1] + + assert coordinator._current_topic_id is None + assert coordinator._current_paper_id is None + assert coordinator._current_reference_papers == [] + assert saved_states[-1]["current_tier"] == "tier1_aggregation" + assert saved_states[-1]["paper_phase"] == "topic_exploration" + assert_proofs_only_never_enters_paper_phase(observation) + + +@pytest.mark.asyncio +async def test_autonomous_provider_credit_pause_preserves_checkpoint_and_resumes(monkeypatch, tmp_path): + events = EventCollector() + fake_stage = FakeProofStage( + pause=True, + checkpoint={ + "source_type": "brainstorm", + "source_id": "topic-paused", + "trigger": "automatic", + "status": "phase_a_running", + "candidates": [ + { + "index": 1, + "candidate": { + "theorem_id": "paused_candidate", + "statement": "A direct proof target from the source.", + "expected_novelty_tier": "novel_variant", + }, + }, + { + "index": 4, + "candidate": { + "theorem_id": "already_processed", + "statement": "A completed proof target.", + "expected_novelty_tier": "novel_variant", + }, + } + ], + "processed_candidate_ids": ["already_processed"], + "attempts_by_candidate": { + "paused_candidate": [ + { + "attempt": 2, + "theorem_id": "paused_candidate", + "reasoning": "Retry with a more direct route.", + "lean_code": "example : True := by trivial", + "error_output": "previous tactic failed", + "strategy": "full_script", + "success": False, + } + ], + "already_processed": [ + { + "attempt": 1, + "theorem_id": "already_processed", + "strategy": "full_script", + "success": True, + } + ], + }, + "theorem_names_by_candidate": { + "paused_candidate": "pausedTheorem", + "already_processed": "finishedTheorem", + }, + "proof_round_index": 1, + }, + ) + fake_metadata = FakeResearchMetadata() + coordinator = AutonomousCoordinator() + coordinator._proof_verification_stage = fake_stage + coordinator._allow_mathematical_proofs = True + coordinator._allow_research_papers = False + coordinator._high_param_model = "rigor-model" + coordinator._high_param_context = 3000 + coordinator._high_param_max_tokens = 300 + coordinator._validator_model = "validator-model" + coordinator._validator_context = 2000 + coordinator._validator_max_tokens = 200 + coordinator._user_research_prompt = "Solve the prompt." + coordinator._state.current_tier = "tier2_paper_writing" + coordinator._broadcast = events.broadcast + + old_lean_enabled = system_config.lean4_enabled + monkeypatch.setattr(coordinator_module, "research_metadata", fake_metadata) + paused_calls = CallRecorder() + monkeypatch.setattr(coordinator_module, "mark_provider_paused", paused_calls) + monkeypatch.setattr(coordinator_module, "wait_for_provider_resume", AsyncMock(return_value=None)) + monkeypatch.setattr( + coordinator_module.proof_database, + "inject_into_prompt", + lambda prompt, **_kwargs: prompt, + raising=False, + ) + monkeypatch.setattr( + coordinator_module.final_answer_memory, + "get_state", + lambda: type("Tier3State", (), {"is_active": False})(), + ) + monkeypatch.setattr(coordinator_module.final_answer_memory, "get_answer_format", lambda: None) + system_config.lean4_enabled = True + try: + status = await coordinator._run_proof_verification( + "Brainstorm source content.", + "brainstorm", + "topic-paused", + source_title="Paused topic", + theorem_candidates=[ + ProofCandidate( + theorem_id="initial_candidate", + statement="A prompt-relevant theorem.", + expected_novelty_tier="novel_variant", + ) + ], + ) + finally: + system_config.lean4_enabled = old_lean_enabled + + observation = RealWorkflowObservation( + runtime_root=tmp_path, + mode=WorkflowMode.AUTONOMOUS, + phase=WorkflowPhase.BRAINSTORM_PROOF, + allow_mathematical_proofs=True, + allow_research_papers=False, + ) + observation.record("autonomous_provider_credit_pause") + for event_type, payload in events.events: + observation.ingest_event(event_type, payload) + + assert status == "complete" + assert fake_stage.pause_count == 1 + assert len(fake_stage.calls) == 2 + assert fake_metadata.saved_proof_checkpoints + saved_checkpoint = fake_metadata.saved_proof_checkpoints[0] + assert saved_checkpoint["source_type"] == "brainstorm" + assert saved_checkpoint["source_id"] == "topic-paused" + assert saved_checkpoint["trigger"] == "automatic" + assert saved_checkpoint["candidates"][0]["candidate"]["theorem_id"] == "paused_candidate" + retry_call = fake_stage.calls[1] + assert [candidate.theorem_id for candidate in retry_call["theorem_candidates"]] == [ + "paused_candidate" + ] + assert retry_call["proof_candidate_indexes"] == { + "paused_candidate": 1, + "already_processed": 4, + } + assert retry_call["checkpoint_attempts_by_candidate"]["paused_candidate"][0].attempt == 2 + assert retry_call["checkpoint_attempts_by_candidate"]["paused_candidate"][0].error_output == ( + "previous tactic failed" + ) + assert retry_call["checkpoint_attempts_by_candidate"]["already_processed"][0].success is True + assert retry_call["checkpoint_theorem_names_by_candidate"] == { + "paused_candidate": "pausedTheorem", + "already_processed": "finishedTheorem", + } + assert retry_call["trigger"] == "automatic" + assert fake_metadata.proof_checkpoint is not None + assert fake_metadata.proof_checkpoint["source_type"] == "brainstorm" + assert fake_metadata.proof_checkpoint["source_id"] == "topic-paused" + assert fake_metadata.proof_checkpoint["trigger"] == "automatic" + assert fake_metadata.proof_checkpoint["status"] == "trigger_complete" + assert fake_metadata.completed_triggers == ["automatic"] + assert fake_metadata.workflow_states[-1]["paper_phase"] == "brainstorm_proof_verification" + paused_payloads = events.payloads("autonomous_proof_provider_paused") + resumed_payloads = events.payloads("autonomous_proof_provider_resumed") + assert paused_payloads + assert resumed_payloads + assert paused_payloads[0]["reason"] == "openrouter_credit_exhaustion" + assert paused_calls.count == 1 + assert_event_sequence( + events, + "autonomous_proof_provider_paused", + "autonomous_proof_provider_resumed", + predicate=lambda payload: payload.get("reason") == "openrouter_credit_exhaustion", + ) + assert observation.provider.credit_exhausted is False + observation.observed_invariants.add("provider.pause_preserves_checkpoint") + assert_observed_invariant(observation, "provider.pause_preserves_checkpoint") + + +@pytest.mark.asyncio +async def test_research_metadata_proof_checkpoint_roundtrip_preserves_resume_cursor(tmp_path): + workflow_path = tmp_path / "workflow_state.json" + checkpoint = { + "source_type": "brainstorm", + "source_id": "topic-roundtrip", + "source_title": "Roundtrip topic", + "trigger": "manual_retry", + "status": "phase_a_running", + "candidates": [ + { + "index": 7, + "candidate": { + "theorem_id": "roundtrip_candidate", + "statement": "A persisted theorem target.", + "expected_novelty_tier": "novel_variant", + }, + } + ], + "processed_candidate_ids": ["finished_candidate"], + "attempts_by_candidate": { + "roundtrip_candidate": [ + { + "attempt": 3, + "theorem_id": "roundtrip_candidate", + "error_output": "persisted Lean failure", + "strategy": "tactic_script", + "success": False, + } + ] + }, + "theorem_names_by_candidate": { + "roundtrip_candidate": "roundtripTheorem", + }, + } + + metadata = ResearchMetadata() + metadata._workflow_state_path = workflow_path + await metadata.save_proof_checkpoint(dict(checkpoint)) + + restored = ResearchMetadata() + restored._workflow_state_path = workflow_path + loaded = await restored.get_proof_checkpoint( + "brainstorm", + "topic-roundtrip", + "manual_retry", + ) + + assert loaded is not None + assert loaded["candidates"] == checkpoint["candidates"] + assert loaded["processed_candidate_ids"] == ["finished_candidate"] + assert loaded["attempts_by_candidate"] == checkpoint["attempts_by_candidate"] + assert loaded["theorem_names_by_candidate"] == checkpoint["theorem_names_by_candidate"] + assert loaded["trigger"] == "manual_retry" + assert loaded["completed_triggers"] == [] + assert loaded["updated_at"] + assert await restored.get_proof_checkpoint("paper", "topic-roundtrip") is None + + await restored.mark_proof_checkpoint_trigger_complete( + "brainstorm", + "topic-roundtrip", + "manual_retry", + "Roundtrip topic", + ) + reloaded = ResearchMetadata() + reloaded._workflow_state_path = workflow_path + completed = await reloaded.get_proof_checkpoint( + "brainstorm", + "topic-roundtrip", + "manual_retry", + ) + assert completed is not None + assert completed["status"] == "trigger_complete" + assert completed["completed_triggers"] == ["manual_retry"] + assert completed["attempts_by_candidate"] == checkpoint["attempts_by_candidate"] + assert completed["theorem_names_by_candidate"] == checkpoint["theorem_names_by_candidate"] + + +@pytest.mark.asyncio +async def test_autonomous_provider_pause_stop_reset_restart_uses_real_event_and_metadata( + monkeypatch, + tmp_path, +): + workflow_path = tmp_path / "workflow_state.json" + metadata = ResearchMetadata() + metadata._workflow_state_path = workflow_path + paused_stage = FakeProofStage( + pause=True, + checkpoint={ + "source_type": "brainstorm", + "source_id": "topic-restart", + "source_title": "Restart topic", + "trigger": "restart_trigger", + "status": "phase_a_running", + "candidates": [ + { + "index": 3, + "candidate": { + "theorem_id": "restart_candidate", + "statement": "A restart-safe proof target.", + "expected_novelty_tier": "novel_variant", + }, + } + ], + "processed_candidate_ids": [], + "attempts_by_candidate": { + "restart_candidate": [ + { + "attempt": 1, + "theorem_id": "restart_candidate", + "error_output": "credit pause after first attempt", + "strategy": "full_script", + "success": False, + } + ] + }, + "theorem_names_by_candidate": {"restart_candidate": "restartTheorem"}, + }, + ) + first = AutonomousCoordinator() + first._proof_verification_stage = paused_stage + first._allow_mathematical_proofs = True + first._high_param_model = "rigor-model" + first._high_param_context = 3000 + first._high_param_max_tokens = 300 + first._validator_model = "validator-model" + first._validator_context = 2000 + first._validator_max_tokens = 200 + first._user_research_prompt = "Solve the restart prompt." + first._base_user_research_prompt = "Solve the restart prompt." + first._state.current_tier = "tier2_paper_writing" + first._current_topic_id = "topic-restart" + first._broadcast = EventCollector().broadcast + + monkeypatch.setattr(coordinator_module, "research_metadata", metadata) + monkeypatch.setattr( + coordinator_module.proof_database, + "inject_into_prompt", + lambda prompt, **_kwargs: prompt, + raising=False, + ) + monkeypatch.setattr( + coordinator_module.final_answer_memory, + "get_state", + lambda: type("Tier3State", (), {"is_active": False})(), + ) + monkeypatch.setattr(coordinator_module.final_answer_memory, "get_answer_format", lambda: None) + old_lean_enabled = system_config.lean4_enabled + system_config.lean4_enabled = True + resume_provider_pauses() + try: + paused_task = asyncio.create_task( + first._run_proof_verification( + "Persisted brainstorm source.", + "brainstorm", + "topic-restart", + source_title="Restart topic", + trigger="restart_trigger", + ) + ) + for _ in range(100): + if paused_stage.pause_count: + break + await asyncio.sleep(0.01) + assert paused_stage.pause_count == 1 + + first._stop_event.set() + with pytest.raises(asyncio.CancelledError): + await asyncio.wait_for(paused_task, timeout=2.0) + persisted = await metadata.get_proof_checkpoint( + "brainstorm", + "topic-restart", + "restart_trigger", + ) + assert persisted is not None + assert persisted["status"] == "phase_a_running" + assert persisted["attempts_by_candidate"]["restart_candidate"][0]["attempt"] == 1 + + assert resume_provider_pauses() >= 1 + restored_metadata = ResearchMetadata() + restored_metadata._workflow_state_path = workflow_path + resumed_stage = FakeProofStage() + restarted = AutonomousCoordinator() + restarted._proof_verification_stage = resumed_stage + restarted._allow_mathematical_proofs = True + restarted._high_param_model = "rigor-model" + restarted._high_param_context = 3000 + restarted._high_param_max_tokens = 300 + restarted._validator_model = "validator-model" + restarted._validator_context = 2000 + restarted._validator_max_tokens = 200 + restarted._user_research_prompt = "Solve the restart prompt." + restarted._base_user_research_prompt = "Solve the restart prompt." + restarted._broadcast = EventCollector().broadcast + monkeypatch.setattr(coordinator_module, "research_metadata", restored_metadata) + + assert await restarted._run_proof_verification( + "Persisted brainstorm source.", + "brainstorm", + "topic-restart", + source_title="Restart topic", + trigger="restart_trigger", + ) == "complete" + assert len(resumed_stage.calls) == 1 + resumed_call = resumed_stage.calls[0] + assert [candidate.theorem_id for candidate in resumed_call["theorem_candidates"]] == [ + "restart_candidate" + ] + assert resumed_call["proof_candidate_indexes"] == {"restart_candidate": 3} + assert resumed_call["checkpoint_attempts_by_candidate"]["restart_candidate"][0].attempt == 1 + assert resumed_call["checkpoint_theorem_names_by_candidate"] == { + "restart_candidate": "restartTheorem" + } + assert resumed_call["trigger"] == "restart_trigger" + completed = await restored_metadata.get_proof_checkpoint( + "brainstorm", + "topic-restart", + "restart_trigger", + ) + assert completed is not None + assert completed["status"] == "trigger_complete" + assert "restart_trigger" in completed["completed_triggers"] + finally: + resume_provider_pauses() + system_config.lean4_enabled = old_lean_enabled + + +@pytest.mark.asyncio +async def test_leanoj_master_proof_stop_resume_preserves_isolated_state(monkeypatch, tmp_path): + request = _leanoj_request() + master_proof = "import Mathlib\n\nexample : 1 = 1 := by\n rfl" + autonomous_proof_store_accessed = False + + class AutonomousProofStoreGuard: + def __getattr__(self, name): + nonlocal autonomous_proof_store_accessed + autonomous_proof_store_accessed = True + raise AssertionError(f"LeanOJ stop/resume must not access autonomous proof storage: {name}") + + async def no_broadcast(*_args, **_kwargs): + return None + + monkeypatch.setattr(system_config, "data_dir", tmp_path) + monkeypatch.setattr(coordinator_module, "proof_database", AutonomousProofStoreGuard()) + + coordinator = LeanOJCoordinator() + monkeypatch.setattr(coordinator, "_broadcast", no_broadcast) + await coordinator.initialize(request) + coordinator._state.phase = "final_proof_loop" + await coordinator._write_master_proof(master_proof, summary="durable final-proof draft") + await coordinator.stop() + + restored = LeanOJCoordinator() + monkeypatch.setattr(restored, "_broadcast", no_broadcast) + resumed = await restored.resume_or_initialize(request) + + observation = RealWorkflowObservation( + runtime_root=tmp_path, + mode=WorkflowMode.LEANOJ, + phase=WorkflowPhase.PAPER_WRITING, + ) + observation.record("leanoj_master_proof_stop_resume") + + assert resumed is True + assert restored.get_state().phase == "final_proof_loop" + assert restored.get_state().master_proof_initialized is True + assert await restored._read_master_proof() == master_proof + assert autonomous_proof_store_accessed is False + assert observation.autonomous_proofs == set() + + +@pytest.mark.asyncio +async def test_leanoj_persisted_provider_pause_resumes_before_nonfinal_workflow( + monkeypatch, + tmp_path, +): + request = _leanoj_request() + events = EventCollector() + monkeypatch.setattr(system_config, "data_dir", tmp_path) + resume_provider_pauses() + + coordinator = LeanOJCoordinator() + coordinator._broadcast = events.broadcast + await coordinator.initialize(request) + coordinator._state.phase = "initial_brainstorm" + coordinator._state.last_active_phase = "initial_brainstorm" + coordinator._state.provider_paused = True + coordinator._state.provider_pause_reason = "openrouter_credit_exhaustion" + coordinator._state.provider_pause_role_id = "leanoj_brainstorm_sub1" + coordinator._state.provider_pause_message = "OpenRouter credits exhausted" + await coordinator._persist_state() + + restored = LeanOJCoordinator() + restored._broadcast = events.broadcast + assert await restored.resume_or_initialize(request) is True + assert restored.get_state().phase == "initial_brainstorm" + assert restored.get_state().provider_paused is True + + entered_phases: list[str] = [] + + async def nonfinal_workflow(_request): + entered_phases.append(restored.get_state().phase) + + monkeypatch.setattr(restored, "_run_workflow", nonfinal_workflow) + start_task = asyncio.create_task(restored.start()) + try: + for _ in range(100): + if events.payloads("leanoj_provider_paused"): + break + await asyncio.sleep(0.01) + assert events.payloads("leanoj_provider_paused") + assert entered_phases == [] + assert resume_provider_pauses() >= 1 + await asyncio.wait_for(start_task, timeout=2.0) + finally: + resume_provider_pauses() + if not start_task.done(): + start_task.cancel() + await asyncio.gather(start_task, return_exceptions=True) + + assert entered_phases == ["initial_brainstorm"] + assert restored.get_state().provider_paused is False + assert restored.get_state().phase != "final_proof_loop" + assert_event_sequence( + events, + "leanoj_provider_paused", + "leanoj_provider_resumed", + predicate=lambda payload: payload.get("reason") == "openrouter_credit_exhaustion", + ) diff --git a/tests/workflow_real_adapters/test_route_conflicts.py b/tests/workflow_real_adapters/test_route_conflicts.py new file mode 100644 index 0000000..c544fc3 --- /dev/null +++ b/tests/workflow_real_adapters/test_route_conflicts.py @@ -0,0 +1,135 @@ +from __future__ import annotations + +from types import SimpleNamespace + +import pytest + +from backend.api.routes import aggregator as aggregator_route +from backend.api.routes import autonomous as autonomous_route +from backend.api.routes import compiler as compiler_route +from backend.api.routes import leanoj as leanoj_route +from backend.shared.workflow_start_guard import WorkflowStartGuard +from tests.workflow_real_adapters.coverage_records import ROUTE_CONFLICT_COVERAGE +from tests.workflow_real_adapters.helpers import inactive_workflow_flags, patch_attributes +from tests.workflow_harness.invariants import assert_single_active_workflow +from tests.workflow_harness.model import WorkflowMode, WorkflowPhase +from tests.workflow_harness.real_adapters import RealWorkflowObservation, route_workflow_state + + +class _RunningFlag: + def __init__(self, is_running: bool = False) -> None: + self.is_running = is_running + + +class _ActiveFlag: + def __init__(self, is_active: bool = False) -> None: + self.is_active = is_active + + +class _AutonomousFlag: + def __init__(self, *, state_running: bool = False, active: bool = False) -> None: + self.is_active = active + self._state_running = state_running + + def get_state(self): + return route_workflow_state(is_running=self._state_running, current_tier="tier1_aggregation") + + +@pytest.mark.parametrize( + ("route_module", "route_label"), + [ + (aggregator_route, "Aggregator"), + (compiler_route, "Compiler"), + (autonomous_route, "Autonomous Research"), + (leanoj_route, "Proof Solver"), + ], +) +@pytest.mark.parametrize( + ("active_mode", "expected_fragment"), + [ + (WorkflowMode.MANUAL_AGGREGATOR, "Aggregator"), + (WorkflowMode.MANUAL_COMPILER, "Compiler"), + (WorkflowMode.AUTONOMOUS, "Autonomous Research"), + (WorkflowMode.LEANOJ, "Proof Solver"), + ], +) +def test_real_route_start_conflicts_preserve_single_workflow_invariant( + monkeypatch, + tmp_path, + route_module, + route_label, + active_mode, + expected_fragment, +): + observation = RealWorkflowObservation( + runtime_root=tmp_path, + mode=active_mode, + phase=WorkflowPhase.TIER1_AGGREGATION if active_mode is WorkflowMode.AUTONOMOUS else WorkflowPhase.PAPER_WRITING, + ) + observation.record("route_start_conflict", route=route_module.__name__, active_mode=active_mode.value) + + monkeypatch.setattr(route_module, "coordinator", _RunningFlag(active_mode is WorkflowMode.MANUAL_AGGREGATOR)) + monkeypatch.setattr(route_module, "compiler_coordinator", _RunningFlag(active_mode is WorkflowMode.MANUAL_COMPILER)) + monkeypatch.setattr( + route_module, + "autonomous_coordinator", + _AutonomousFlag( + state_running=active_mode is WorkflowMode.AUTONOMOUS, + active=active_mode is WorkflowMode.AUTONOMOUS, + ), + ) + monkeypatch.setattr(route_module, "leanoj_coordinator", _ActiveFlag(active_mode is WorkflowMode.LEANOJ)) + + conflict = route_module._get_start_conflict() + + if route_label == expected_fragment: + assert conflict is not None + assert "already running" in conflict + else: + assert conflict is not None + assert expected_fragment in conflict + observation.emit("start_blocked", active_mode=active_mode.value, route=route_module.__name__) + assert_single_active_workflow(observation) + + +def test_real_autonomous_route_counts_pending_child_activity_as_active(monkeypatch, tmp_path): + observation = RealWorkflowObservation( + runtime_root=tmp_path, + mode=WorkflowMode.AUTONOMOUS, + phase=WorkflowPhase.TIER1_AGGREGATION, + ) + observation.record("autonomous_child_pending_conflict") + + route_flags = inactive_workflow_flags() + route_flags["autonomous_coordinator"] = SimpleNamespace( + is_active=True, + get_state=lambda: route_workflow_state(is_running=False, current_tier="tier1_aggregation"), + ) + patch_attributes(monkeypatch, autonomous_route, route_flags) + + conflict = autonomous_route._get_start_conflict() + + assert conflict == "Autonomous research is already running" + observation.emit("start_blocked", active_mode=WorkflowMode.AUTONOMOUS.value) + assert_single_active_workflow(observation) + + +def test_compiler_proof_only_guard_owner_blocks_every_other_route(monkeypatch): + guard = WorkflowStartGuard() + monkeypatch.setattr( + "backend.shared.workflow_start_guard.sleep_inhibitor.acquire", + lambda _owner: None, + ) + guard.commit(compiler_route.COMPILER_PROOF_ONLY_OWNER) + for module in ( + aggregator_route, + autonomous_route, + compiler_route, + leanoj_route, + ): + monkeypatch.setattr(module, "workflow_start_guard", guard) + + assert aggregator_route._get_start_conflict() + assert autonomous_route._get_start_conflict() + assert compiler_route._get_start_conflict() + assert leanoj_route._get_start_conflict() diff --git a/tests/workflow_recombinatory/test_deep_workflow_state_machine.py b/tests/workflow_recombinatory/test_deep_workflow_state_machine.py new file mode 100644 index 0000000..d2b917e --- /dev/null +++ b/tests/workflow_recombinatory/test_deep_workflow_state_machine.py @@ -0,0 +1,53 @@ +from __future__ import annotations + +import os + +import pytest + +from tests.workflow_harness.exercise_evidence import assert_exercise_tokens +from tests.workflow_harness.model import WorkflowModel +from tests.workflow_harness.recombinatory_generator import ( + DEFAULT_TARGETS, + run_generated_scenario, +) + + +pytestmark = pytest.mark.skipif( + os.getenv("MOTO_WORKFLOW_DEEP_TESTS") != "1", + reason="set MOTO_WORKFLOW_DEEP_TESTS=1 to run deep workflow tests", +) + +DEEP_SEEDS = (3, 7, 41, 53, 73, 89, 149, 211, 307, 401) + + +@pytest.mark.parametrize("target", DEFAULT_TARGETS, ids=lambda target: target.target_id) +@pytest.mark.parametrize("seed", DEEP_SEEDS) +def test_deep_seed_matrix_preserves_invariants_coverage_and_replay( + tmp_path, target, seed +) -> None: + first_model = WorkflowModel( + runtime_root=tmp_path / target.target_id / f"seed-{seed}" / "first" + ) + first = run_generated_scenario( + first_model, + target, + seed=seed, + ) + replayed = run_generated_scenario( + WorkflowModel(runtime_root=tmp_path / target.target_id / f"seed-{seed}" / "replay"), + target, + seed=seed, + ) + + assert first.action_count <= target.max_steps + assert first.fields == target.fields + assert set(target.fields).issubset(first.observed_fields) + assert first.invariants == target.invariants + assert set(target.invariants) == first.exercised_invariants + assert_exercise_tokens( + first_model, + target.must_exercise, + scenario_id=target.target_id, + ) + assert first.action_ids == replayed.action_ids + assert first.replay == replayed.replay diff --git a/tests/workflow_recombinatory/test_mode_aware_generator.py b/tests/workflow_recombinatory/test_mode_aware_generator.py new file mode 100644 index 0000000..4703494 --- /dev/null +++ b/tests/workflow_recombinatory/test_mode_aware_generator.py @@ -0,0 +1,592 @@ +from __future__ import annotations + +from pathlib import Path + +import pytest + +from tests.workflow_harness import actions +from tests.workflow_harness.recombinatory_generator import ( + ACTION_SPECS, + DEFAULT_TARGETS, + ActionSpec, + GeneratedRunFailure, + GenerationTarget, + REPLAY_NOT_REPRODUCED, + ReplayRejected, + action_specs_by_id, + eligible_action_specs, + format_failure, + reduce_failing_action_ids, + replay_action_ids, + run_generated_scenario, + validate_action_specs, + validate_generation_target, +) +from tests.workflow_harness.exercise_evidence import observed_exercise_tokens +from tests.workflow_harness.model import WorkflowEvent, WorkflowMode, WorkflowModel + + +def _target(target_id: str) -> GenerationTarget: + return next(target for target in DEFAULT_TARGETS if target.target_id == target_id) + + +def test_same_seed_and_target_produce_same_action_ids_and_replay(tmp_path): + target = _target("generated_allowed_outputs_provider_pause_resume") + first = run_generated_scenario( + WorkflowModel(runtime_root=tmp_path / "first"), + target, + seed=41, + ) + second = run_generated_scenario( + WorkflowModel(runtime_root=tmp_path / "second"), + target, + seed=41, + ) + + assert first.action_ids == second.action_ids + assert first.replay == second.replay + + +def test_fixed_seed_matrix_preserves_shortest_path_even_when_seed_varies(tmp_path): + target = _target("generated_assistant_prompt_context") + runs = { + run_generated_scenario( + WorkflowModel(runtime_root=tmp_path / f"seed-{seed}"), + target, + seed=seed, + ).action_ids + for seed in (7, 19, 41, 73, 101) + } + + assert runs == { + ( + "start_autonomous_papers_and_proofs", + "refresh_assistant_pack", + "prepare_prompt_context", + "disable_session_history", + ) + } + + +def test_eligible_actions_are_state_aware_and_do_not_include_no_ops(tmp_path): + model = WorkflowModel(runtime_root=tmp_path) + idle_ids = {spec.action_id for spec in eligible_action_specs(model)} + + assert "start_autonomous_proofs_only" in idle_ids + assert "complete_brainstorm" not in idle_ids + assert "resume" not in idle_ids + + actions.start_autonomous_proofs_only(model) + active_ids = {spec.action_id for spec in eligible_action_specs(model)} + + assert "complete_topic_exploration" in active_ids + assert "start_autonomous_proofs_only" not in active_ids + assert "attempt_conflicting_start" in active_ids + + +def test_default_registry_and_targets_are_valid(): + validate_action_specs() + for target in DEFAULT_TARGETS: + validate_generation_target(target) + + +def test_invalid_registry_field_is_rejected(): + invalid = ( + ActionSpec( + action_id="invalid", + execute=actions.stop, + fields=("not_a_field",), + eligible_when=lambda model: True, + ), + ) + + with pytest.raises(AssertionError, match="unknown fields"): + validate_action_specs(invalid) + + +def test_target_with_duplicate_fields_is_rejected(): + target = GenerationTarget( + target_id="duplicate_fields", + fields=("runtime_exclusivity", "runtime_exclusivity"), + invariants=("runtime.child_tasks_count_as_active",), + must_exercise=(), + ) + + with pytest.raises(AssertionError, match="at least two unique fields"): + validate_generation_target(target) + + +def test_missing_required_evidence_reports_generation_metadata(tmp_path): + target = GenerationTarget( + target_id="impossible_in_one_step", + fields=("provider_pause_resume", "workflow_filesystem_state"), + invariants=("provider.pause_preserves_checkpoint",), + must_exercise=("provider_pause",), + max_steps=1, + ) + + with pytest.raises(GeneratedRunFailure) as error: + run_generated_scenario( + WorkflowModel(runtime_root=tmp_path), + target, + seed=41, + ) + + message = str(error.value) + assert "Seed: 41" in message + assert "Fields: provider_pause_resume x workflow_filesystem_state" in message + assert "Action count: 0" in message + assert "Invariant: generator.target_unreachable" in message + assert "Replay:" in message + + +def test_failure_formatter_contains_numbered_replay(): + message = format_failure( + seed=7, + fields=("allowed_outputs", "proof_runtime_gating"), + action_count=2, + invariant="outputs.at_least_one_output_enabled", + replay=("first()", "second()"), + detail="failed", + ) + + assert "Seed: 7" in message + assert "Fields: allowed_outputs x proof_runtime_gating" in message + assert "Action count: 2" in message + assert "Invariant: outputs.at_least_one_output_enabled" in message + assert "1. first()" in message + assert "2. second()" in message + + +def test_every_default_target_can_complete_for_normal_seed(tmp_path): + for target in DEFAULT_TARGETS: + result = run_generated_scenario( + WorkflowModel(runtime_root=tmp_path / target.target_id), + target, + seed=41, + ) + assert set(target.must_exercise).issubset(result.observed_evidence) + assert set(target.fields).issubset(result.observed_fields) + assert set(target.invariants) == result.exercised_invariants + assert result.action_count <= target.max_steps + assert result.action_ids + + +@pytest.mark.parametrize( + ("target_id", "required_actions"), + [ + ( + "generated_manual_aggregator_lifecycle", + { + "start_manual_aggregator", + "accept_manual_aggregator_submission", + "clear_manual_aggregator_state", + }, + ), + ( + "generated_leanoj_durable_lifecycle", + { + "start_leanoj", + "edit_leanoj_master_proof", + "skip_leanoj_brainstorm", + "force_leanoj_brainstorm", + "stop", + "resume", + "clear_leanoj_state", + }, + ), + ( + "generated_autonomous_paper_checkpoint", + { + "start_autonomous_papers_and_proofs", + "attempt_conflicting_start", + "complete_topic_exploration", + "force_paper_writing", + "enter_autonomous_paper_checkpoint", + "complete_autonomous_paper_checkpoint", + }, + ), + ], +) +def test_representative_new_action_families_are_generated( + tmp_path, target_id, required_actions +): + result = run_generated_scenario( + WorkflowModel(runtime_root=tmp_path / target_id), + _target(target_id), + seed=41, + ) + assert required_actions.issubset(result.action_ids) + + +def test_all_action_specs_have_stable_unique_ids(): + ids = [spec.action_id for spec in ACTION_SPECS] + assert ids == list(dict.fromkeys(ids)) + + +def _replay_fixture(): + def start(model): + model.record("fixture_start") + model.mode = WorkflowMode.AUTONOMOUS + + def noise_one(model): + model.record("fixture_noise_one") + + def noise_two(model): + model.record("fixture_noise_two") + + def trigger(model): + model.record("fixture_trigger") + model.terminal_stop_events += 1 + + idle = lambda model: model.mode is WorkflowMode.NONE + active = lambda model: model.mode is WorkflowMode.AUTONOMOUS + specs = ( + ActionSpec("fixture.start", start, ("runtime_exclusivity",), idle), + ActionSpec("fixture.noise_one", noise_one, ("runtime_exclusivity",), active), + ActionSpec("fixture.noise_two", noise_two, ("runtime_exclusivity",), active), + ActionSpec("fixture.trigger", trigger, ("runtime_exclusivity",), active), + ) + return specs, lambda: WorkflowModel(runtime_root=Path(".")) + + +def test_action_lookup_and_replay_reproduce_named_failure_category(): + specs, model_factory = _replay_fixture() + + assert tuple(action_specs_by_id(specs)) == tuple(spec.action_id for spec in specs) + result = replay_action_ids( + model_factory(), + ("fixture.start", "fixture.noise_one", "fixture.trigger"), + action_specs=specs, + failure_predicate=lambda model: model.terminal_stop_events == 1, + failure_category="fixture.terminal_stop", + ) + + assert result.failed + assert result.failure_category == "fixture.terminal_stop" + assert result.action_ids == ( + "fixture.start", + "fixture.noise_one", + "fixture.trigger", + ) + assert result.replay[-1] == "fixture_trigger()" + + +def test_replay_rejects_unknown_and_state_ineligible_action_ids(): + specs, model_factory = _replay_fixture() + + with pytest.raises(ReplayRejected, match="Unknown action ID"): + replay_action_ids(model_factory(), ("fixture.unknown",), action_specs=specs) + with pytest.raises(ReplayRejected, match="ineligible"): + replay_action_ids(model_factory(), ("fixture.trigger",), action_specs=specs) + + +def test_requested_replay_category_reports_not_reproduced_exactly(): + specs, model_factory = _replay_fixture() + + result = replay_action_ids( + model_factory(), + ("fixture.start", "fixture.noise_one"), + action_specs=specs, + failure_predicate=lambda model: model.terminal_stop_events == 1, + failure_category="fixture.terminal_stop", + ) + + assert result.failure_category == REPLAY_NOT_REPRODUCED + assert not result.failed + assert result.failure_detail == ( + "Requested failure category 'fixture.terminal_stop' was not reproduced." + ) + + +def test_requested_replay_category_does_not_accept_different_invariant_failure(): + specs, model_factory = _replay_fixture() + + def break_invariant(model): + model.record("fixture_break_invariant") + model.prompt_user_direct_injected = False + + breaking_specs = specs + ( + ActionSpec( + "fixture.break_invariant", + break_invariant, + ("runtime_exclusivity",), + lambda model: model.mode is WorkflowMode.AUTONOMOUS, + ), + ) + result = replay_action_ids( + model_factory(), + ("fixture.start", "fixture.break_invariant"), + action_specs=breaking_specs, + failure_category="fixture.terminal_stop", + ) + + assert result.failure_category == REPLAY_NOT_REPRODUCED + assert "observed 'prompt.user_prompt_direct_injected'" in result.failure_detail + + +def test_reducer_finds_shortest_prefix_and_one_minimal_failure(): + specs, model_factory = _replay_fixture() + original = ( + "fixture.start", + "fixture.noise_one", + "fixture.noise_two", + "fixture.trigger", + "fixture.noise_one", + ) + predicate = lambda model: model.terminal_stop_events == 1 + reduction = reduce_failing_action_ids( + model_factory, + original, + action_specs=specs, + failure_predicate=predicate, + failure_category="fixture.terminal_stop", + ) + + assert reduction.shortest_failing_prefix == original[:4] + assert reduction.action_ids == ("fixture.start", "fixture.trigger") + assert reduction.minimized + + for index in range(len(reduction.action_ids)): + candidate = reduction.action_ids[:index] + reduction.action_ids[index + 1 :] + try: + result = replay_action_ids( + model_factory(), + candidate, + action_specs=specs, + failure_predicate=predicate, + failure_category="fixture.terminal_stop", + ) + except ReplayRejected: + continue + assert not result.failed + + +def test_reducer_returns_original_for_successful_replay_without_minimizing(): + specs, model_factory = _replay_fixture() + original = ("fixture.start", "fixture.noise_one") + + reduction = reduce_failing_action_ids( + model_factory, + original, + action_specs=specs, + failure_predicate=lambda model: model.terminal_stop_events == 1, + failure_category="fixture.terminal_stop", + ) + + assert reduction.action_ids == original + assert reduction.failure_category is None + assert not reduction.minimized + + +def test_replay_reduction_is_deterministic(): + specs, model_factory = _replay_fixture() + original = ( + "fixture.start", + "fixture.noise_one", + "fixture.noise_two", + "fixture.trigger", + ) + kwargs = { + "action_specs": specs, + "failure_predicate": lambda model: model.terminal_stop_events == 1, + "failure_category": "fixture.terminal_stop", + } + + assert reduce_failing_action_ids(model_factory, original, **kwargs) == ( + reduce_failing_action_ids(model_factory, original, **kwargs) + ) + + +@pytest.mark.parametrize("seed", range(64)) +def test_every_seed_reaches_every_default_target_within_budget(tmp_path, seed): + for target in DEFAULT_TARGETS: + result = run_generated_scenario( + WorkflowModel(runtime_root=tmp_path / f"{target.target_id}-{seed}"), + target, + seed=seed, + ) + assert set(target.must_exercise).issubset(result.observed_evidence) + assert set(target.fields).issubset(result.observed_fields) + assert result.action_count <= target.max_steps + + +def test_planner_avoids_high_weight_dead_end_action(tmp_path): + target = GenerationTarget( + target_id="dead_end_adversary", + fields=("runtime_exclusivity", "websocket_api_contracts"), + invariants=("runtime.single_active_workflow",), + must_exercise=("start_blocked",), + max_steps=2, + ) + specs = ( + ActionSpec( + action_id="clear_dead_end", + execute=actions.clear, + fields=("runtime_exclusivity", "websocket_api_contracts"), + eligible_when=lambda model: True, + weight=10_000, + ), + next(spec for spec in ACTION_SPECS if spec.action_id == "start_manual_compiler"), + next(spec for spec in ACTION_SPECS if spec.action_id == "attempt_conflicting_start"), + ) + + result = run_generated_scenario( + WorkflowModel(runtime_root=tmp_path), + target, + seed=7, + action_specs=specs, + ) + + assert result.action_ids == ("start_manual_compiler", "attempt_conflicting_start") + + +def test_planner_returns_true_shortest_path_not_depth_first_first_hit(tmp_path): + target = GenerationTarget( + target_id="shortest_path_adversary", + fields=("runtime_exclusivity", "websocket_api_contracts"), + invariants=("runtime.single_active_workflow",), + must_exercise=("start_blocked",), + max_steps=3, + ) + + def enter_long(model): + model.record("enter_long") + model.checkpoint["route"] = "long-start" + + def continue_long(model): + model.record("continue_long") + model.checkpoint["route"] = "long-ready" + + def enter_short(model): + model.record("enter_short") + model.checkpoint["route"] = "short-ready" + + def trigger(model): + model.record("trigger") + model.emit("start_blocked", active_mode="fixture") + + specs = ( + ActionSpec( + "fixture.enter_long", + enter_long, + target.fields, + lambda model: "route" not in model.checkpoint, + weight=10_000, + ), + ActionSpec( + "fixture.continue_long", + continue_long, + target.fields, + lambda model: model.checkpoint.get("route") == "long-start", + ), + ActionSpec( + "fixture.enter_short", + enter_short, + target.fields, + lambda model: "route" not in model.checkpoint, + ), + ActionSpec( + "fixture.trigger", + trigger, + target.fields, + lambda model: model.checkpoint.get("route") in {"long-ready", "short-ready"}, + ), + ) + + result = run_generated_scenario( + WorkflowModel(runtime_root=tmp_path), + target, + seed=7, + action_specs=specs, + ) + + assert result.action_ids == ("fixture.enter_short", "fixture.trigger") + + +@pytest.mark.parametrize("history_kind", ["events", "persisted_events"]) +def test_planning_signature_preserves_relevant_event_history(tmp_path, history_kind): + target = GenerationTarget( + target_id=f"{history_kind}_history_adversary", + fields=("runtime_exclusivity", "websocket_api_contracts"), + invariants=("runtime.single_active_workflow",), + must_exercise=("start_blocked",), + max_steps=2, + ) + + def record_dead_history(model): + model.record("record_dead_history") + getattr(model, history_kind).append(WorkflowEvent("dead_history")) + + def record_live_history(model): + model.record("record_live_history") + getattr(model, history_kind).append(WorkflowEvent("live_history")) + + def trigger(model): + model.record("history_trigger") + model.emit("start_blocked", active_mode="fixture") + + def live_history(model): + return any( + event.event_type == "live_history" for event in getattr(model, history_kind) + ) + + specs = ( + ActionSpec("fixture.dead", record_dead_history, target.fields, lambda model: True), + ActionSpec("fixture.live", record_live_history, target.fields, lambda model: True), + ActionSpec("fixture.trigger", trigger, target.fields, live_history), + ) + + result = run_generated_scenario( + WorkflowModel(runtime_root=tmp_path), + target, + seed=19, + action_specs=specs, + ) + + assert result.action_ids == ("fixture.live", "fixture.trigger") + + +def test_build_e_evidence_requires_concrete_observations_not_default_flags(tmp_path): + model = WorkflowModel(runtime_root=tmp_path) + + assert model.prompt_user_direct_injected + assert model.direct_sources_excluded_from_rag + assert model.mandatory_source_overflow_visible + assert not { + "prompt_context", + "rag_source_exclusion", + "mandatory_source_overflow", + }.intersection(observed_exercise_tokens(model)) + + model.prepare_prompt_context() + model.verify_rag_source_exclusion() + model.reject_mandatory_source_overflow() + + assert { + "prompt_context", + "rag_source_exclusion", + "mandatory_source_overflow", + }.issubset(observed_exercise_tokens(model)) + + +def test_unreachable_target_fails_before_executing_real_model(tmp_path): + target = GenerationTarget( + target_id="unreachable_adversary", + fields=("provider_pause_resume", "workflow_filesystem_state"), + invariants=("provider.pause_preserves_checkpoint",), + must_exercise=("provider_pause",), + max_steps=1, + ) + model = WorkflowModel(runtime_root=tmp_path) + + with pytest.raises(GeneratedRunFailure) as error: + run_generated_scenario(model, target, seed=73) + + message = str(error.value) + assert "Action count: 0" in message + assert "Invariant: generator.target_unreachable" in message + assert "No path within 1 steps" in message + assert "missing evidence ['provider_pause']" in message + assert "visited " in message + assert model.replay == [] diff --git a/tests/workflow_recombinatory/test_workflow_state_machine.py b/tests/workflow_recombinatory/test_workflow_state_machine.py new file mode 100644 index 0000000..3b84e4d --- /dev/null +++ b/tests/workflow_recombinatory/test_workflow_state_machine.py @@ -0,0 +1,30 @@ +from __future__ import annotations + +import pytest + +from tests.workflow_harness.exercise_evidence import assert_exercise_tokens +from tests.workflow_harness.recombinatory_generator import ( + DEFAULT_TARGETS, + run_generated_scenario, +) +from tests.workflow_harness.model import WorkflowModel + + +SEEDS = (7, 41) + + +@pytest.mark.parametrize("target", DEFAULT_TARGETS, ids=lambda target: target.target_id) +@pytest.mark.parametrize("seed", SEEDS) +def test_seeded_workflow_action_recombination_preserves_invariants_and_coverage( + tmp_path, target, seed +): + model = WorkflowModel(runtime_root=tmp_path / target.target_id / f"seed-{seed}") + + result = run_generated_scenario(model, target, seed=seed) + + assert result.action_count <= target.max_steps + assert result.fields == target.fields + assert set(target.fields).issubset(result.observed_fields) + assert result.invariants == target.invariants + assert set(target.invariants) == result.exercised_invariants + assert_exercise_tokens(model, target.must_exercise, scenario_id=target.target_id) diff --git a/tests/workflow_scenarios/test_cross_field_analysis.py b/tests/workflow_scenarios/test_cross_field_analysis.py new file mode 100644 index 0000000..bc4ae89 --- /dev/null +++ b/tests/workflow_scenarios/test_cross_field_analysis.py @@ -0,0 +1,90 @@ +from __future__ import annotations + +import pytest + +from tests.workflow_harness.actions import run_actions +from tests.workflow_harness.coverage_metadata import assert_coverage_records_valid +from tests.workflow_harness.cross_field_scenarios import ( + CROSS_FIELD_SCENARIOS, + first_build_cross_field_coverage, + validate_cross_field_scenario, +) +from tests.workflow_harness.exercise_evidence import assert_exercise_tokens +from tests.workflow_harness.invariants import assert_all_invariants +from tests.workflow_harness.model import WorkflowModel, WorkflowPhase + + +TEST_FILE = "tests/workflow_scenarios/test_cross_field_analysis.py" +HIGH_PRIORITY_FIELDS = { + "allowed_outputs", + "provider_pause_resume", + "proof_runtime_gating", + "assistant_memory", + "proof_scope_isolation", + "workflow_filesystem_state", + "websocket_api_contracts", + "prompt_context", +} + + +def test_cross_field_scenario_artifacts_are_valid(): + scenario_ids = [scenario.scenario_id for scenario in CROSS_FIELD_SCENARIOS] + assert len(scenario_ids) == len(set(scenario_ids)) + + for scenario in CROSS_FIELD_SCENARIOS: + validate_cross_field_scenario(scenario) + + +@pytest.mark.parametrize("scenario", CROSS_FIELD_SCENARIOS, ids=lambda scenario: scenario.scenario_id) +def test_model_cross_field_scenarios_preserve_invariants_and_exercise_fields(tmp_path, scenario): + model = WorkflowModel(runtime_root=tmp_path / scenario.scenario_id) + + run_actions(model, list(scenario.actions)) + + assert_all_invariants(model) + assert_exercise_tokens(model, scenario.must_exercise, scenario_id=scenario.scenario_id) + + +def test_first_build_cross_field_coverage_metadata_is_valid(): + records = first_build_cross_field_coverage(TEST_FILE) + + assert_coverage_records_valid(records) + + +def test_first_build_cross_field_coverage_map_includes_high_priority_fields(): + records = first_build_cross_field_coverage(TEST_FILE) + covered_fields = {field for record in records for field in record.fields} + covered_invariants = {invariant for record in records for invariant in record.invariants} + + assert HIGH_PRIORITY_FIELDS.issubset(covered_fields) + assert { + "outputs.proofs_only_no_paper_phase", + "provider.pause_preserves_checkpoint", + "assistant.no_validator_injection", + "proof_scope.manual_clear_archives_active", + "events.frontend_events_include_scope_phase", + "proof_runtime.no_lean_when_disabled", + }.issubset(covered_invariants) + + +def test_first_build_cross_field_scenarios_are_inspectable_without_runtime_dependencies(): + for scenario in CROSS_FIELD_SCENARIOS: + assert scenario.adapter == "model" + assert scenario.expected_result == "passed" + assert not scenario.notes + assert all(callable(action) for action in scenario.actions) + + +def test_proofs_only_cross_field_scenario_does_not_land_in_paper_phase(tmp_path): + scenario = next( + item + for item in CROSS_FIELD_SCENARIOS + if item.scenario_id == "model_allowed_outputs_provider_pause_stop_resume" + ) + model = WorkflowModel(runtime_root=tmp_path / scenario.scenario_id) + + run_actions(model, list(scenario.actions)) + + assert model.phase is WorkflowPhase.TIER1_AGGREGATION + assert model.allow_mathematical_proofs is True + assert model.allow_research_papers is False diff --git a/tests/workflow_scenarios/test_invariant_catalog.py b/tests/workflow_scenarios/test_invariant_catalog.py new file mode 100644 index 0000000..43386da --- /dev/null +++ b/tests/workflow_scenarios/test_invariant_catalog.py @@ -0,0 +1,349 @@ +from __future__ import annotations + +import pytest + +from tests.workflow_harness.actions import ( + attempt_disabled_lean_checkpoint, + assistant_retrieve_non_blocking, + assistant_stagnant_pack_backoff, + clear, + complete_brainstorm, + complete_brainstorm_with_generated_paper, + complete_topic_exploration, + disable_session_history, + emit_context_overflow_contract_events, + emit_frontend_scoped_event, + emit_registered_proof_verified, + force_paper_writing, + prepare_prompt_context, + prepare_validator_prompt_with_live_assistant, + prune_paper, + reject_mandatory_source_overflow, + refresh_assistant_pack, + reset_credit, + resolve_runtime_path, + resume, + resume_completed_brainstorm, + run_actions, + run_manual_proof_check, + run_smt_hint_generation, + simulate_credit_exhaustion, + simulate_provider_output_truncation, + start_autonomous_papers_and_proofs, + start_autonomous_papers_only, + start_autonomous_no_outputs, + start_autonomous_proofs_only, + start_child_task, + start_manual_compiler, + stop, + stale_child_output_arrives_after_parent_action, + try_hosted_desktop_only_route, + verify_rag_source_exclusion, +) +from tests.workflow_harness.invariant_catalog import ( + FIRST_BUILD_FIELDS, + INVARIANT_CATALOG, + INVARIANTS_BY_ID, + assert_invariant, + invariant_ids_for_adapter, +) +from tests.workflow_harness.model import WorkflowEvent, WorkflowMode, WorkflowModel, WorkflowPhase + + +EXPECTED_FIRST_BUILD_FIELDS = { + "runtime_exclusivity", + "proof_runtime_gating", + "allowed_outputs", + "provider_pause_resume", + "assistant_memory", + "proof_scope_isolation", +} + +PLAN2_INVARIANT_IDS = { + "runtime.single_active_workflow", + "runtime.child_tasks_count_as_active", + "runtime.parent_action_fences_child_outputs", + "proof_runtime.no_lean_when_disabled", + "proof_runtime.no_smt_when_disabled", + "proof_runtime.hosted_proof_settings_unavailable", + "proof_runtime.truncation_is_attempt_failure", + "outputs.at_least_one_output_enabled", + "outputs.proofs_only_no_paper_phase", + "outputs.papers_only_skips_proof_work", + "provider.pause_preserves_checkpoint", + "provider.stop_resume_preserves_pause", + "provider.reset_wakes_without_corrupting_checkpoint", + "assistant.no_validator_injection", + "assistant.disable_clears_live_pack", + "assistant.non_blocking_retrieval", + "assistant.stagnant_pack_backoff_not_shutdown", + "proof_scope.manual_not_in_autonomous_current", + "proof_scope.autonomous_events_not_manual", + "proof_scope.manual_clear_archives_active", + "proof_scope.generated_appendices_stripped_from_prompts", + "state.clear_removes_active_preserves_history", + "state.completed_brainstorm_no_replay_handoff", + "state.pruned_papers_excluded_from_context", + "state.runtime_roots_are_active_roots", + "events.frontend_events_include_scope_phase", + "events.context_overflow_route_identity", + "events.context_overflow_persists_across_reload", + "events.context_overflow_terminal_stop_once", + "events.proof_context_overflow_nonfatal", + "events.proof_verified_after_registration", + "api.hosted_desktop_only_routes_unavailable", + "prompt.user_prompt_direct_injected", + "prompt.validator_excludes_assistant_memory", + "prompt.proof_source_context_required", + "prompt.direct_sources_excluded_from_rag", + "prompt.mandatory_source_overflow_fails_visible", + "prompt.generated_appendices_stripped", +} + + +SCENARIOS_BY_INVARIANT = { + "runtime.single_active_workflow": [start_autonomous_papers_and_proofs], + "runtime.child_tasks_count_as_active": [start_autonomous_papers_and_proofs, start_child_task], + "runtime.parent_action_fences_child_outputs": [ + start_autonomous_papers_and_proofs, + complete_topic_exploration, + start_child_task, + force_paper_writing, + stale_child_output_arrives_after_parent_action, + ], + "proof_runtime.no_lean_when_disabled": [ + start_autonomous_papers_only, + attempt_disabled_lean_checkpoint, + ], + "proof_runtime.no_smt_when_disabled": [ + start_autonomous_papers_and_proofs, + run_smt_hint_generation, + ], + "proof_runtime.hosted_proof_settings_unavailable": [try_hosted_desktop_only_route], + "proof_runtime.truncation_is_attempt_failure": [simulate_provider_output_truncation], + "outputs.at_least_one_output_enabled": [start_autonomous_no_outputs], + "outputs.proofs_only_no_paper_phase": [ + start_autonomous_proofs_only, + complete_topic_exploration, + complete_brainstorm, + ], + "outputs.papers_only_skips_proof_work": [ + start_autonomous_papers_only, + complete_topic_exploration, + complete_brainstorm, + ], + "provider.pause_preserves_checkpoint": [ + start_autonomous_papers_and_proofs, + complete_topic_exploration, + simulate_credit_exhaustion, + complete_brainstorm, + stop, + resume, + ], + "provider.stop_resume_preserves_pause": [ + start_autonomous_papers_and_proofs, + complete_topic_exploration, + simulate_credit_exhaustion, + complete_brainstorm, + ], + "provider.reset_wakes_without_corrupting_checkpoint": [ + start_autonomous_papers_and_proofs, + complete_topic_exploration, + simulate_credit_exhaustion, + complete_brainstorm, + reset_credit, + ], + "assistant.no_validator_injection": [ + start_autonomous_papers_and_proofs, + refresh_assistant_pack, + prepare_validator_prompt_with_live_assistant, + ], + "assistant.disable_clears_live_pack": [ + start_autonomous_papers_and_proofs, + refresh_assistant_pack, + disable_session_history, + ], + "assistant.non_blocking_retrieval": [ + start_autonomous_papers_and_proofs, + assistant_retrieve_non_blocking, + ], + "assistant.stagnant_pack_backoff_not_shutdown": [ + start_autonomous_papers_and_proofs, + assistant_stagnant_pack_backoff, + ], + "proof_scope.manual_not_in_autonomous_current": [ + start_manual_compiler, + run_manual_proof_check, + ], + "proof_scope.autonomous_events_not_manual": [ + start_autonomous_papers_and_proofs, + complete_topic_exploration, + emit_registered_proof_verified, + ], + "proof_scope.manual_clear_archives_active": [ + start_manual_compiler, + run_manual_proof_check, + clear, + ], + "proof_scope.generated_appendices_stripped_from_prompts": [prepare_prompt_context], + "state.clear_removes_active_preserves_history": [ + start_manual_compiler, + run_manual_proof_check, + clear, + ], + "state.completed_brainstorm_no_replay_handoff": [ + complete_brainstorm_with_generated_paper, + resume_completed_brainstorm, + ], + "state.pruned_papers_excluded_from_context": [prune_paper], + "state.runtime_roots_are_active_roots": [resolve_runtime_path], + "events.frontend_events_include_scope_phase": [ + start_autonomous_papers_and_proofs, + emit_frontend_scoped_event, + ], + "events.context_overflow_route_identity": [emit_context_overflow_contract_events], + "events.context_overflow_persists_across_reload": [emit_context_overflow_contract_events], + "events.context_overflow_terminal_stop_once": [emit_context_overflow_contract_events], + "events.proof_context_overflow_nonfatal": [emit_context_overflow_contract_events], + "events.proof_verified_after_registration": [ + start_autonomous_papers_and_proofs, + emit_registered_proof_verified, + ], + "api.hosted_desktop_only_routes_unavailable": [try_hosted_desktop_only_route], + "prompt.user_prompt_direct_injected": [prepare_prompt_context], + "prompt.validator_excludes_assistant_memory": [prepare_prompt_context], + "prompt.proof_source_context_required": [prepare_prompt_context], + "prompt.direct_sources_excluded_from_rag": [verify_rag_source_exclusion], + "prompt.mandatory_source_overflow_fails_visible": [reject_mandatory_source_overflow], + "prompt.generated_appendices_stripped": [prepare_prompt_context], +} + + +def _make_invariant_violation(model: WorkflowModel, invariant_id: str) -> None: + model.record("inject_invariant_violation", invariant_id=invariant_id) + if invariant_id == "runtime.single_active_workflow": + model.active_owners = {WorkflowMode.AUTONOMOUS, WorkflowMode.LEANOJ} + elif invariant_id == "runtime.child_tasks_count_as_active": + model.pending_child_tasks = 1 + elif invariant_id == "runtime.parent_action_fences_child_outputs": + model.stale_child_outputs_fenced = False + elif invariant_id == "proof_runtime.no_lean_when_disabled": + model.lean.blocked_invocations = 1 + elif invariant_id == "proof_runtime.no_smt_when_disabled": + model.smt.blocked_invocations = 1 + elif invariant_id == "proof_runtime.hosted_proof_settings_unavailable": + model.hosted_desktop_route_attempted = True + model.hosted_desktop_route_unavailable = False + elif invariant_id == "proof_runtime.truncation_is_attempt_failure": + model.truncation_failures = model.provider_pause_from_truncation = 1 + elif invariant_id == "outputs.at_least_one_output_enabled": + model.allow_mathematical_proofs = model.allow_research_papers = False + elif invariant_id == "outputs.proofs_only_no_paper_phase": + model.mode = WorkflowMode.AUTONOMOUS + model.allow_mathematical_proofs = True + model.allow_research_papers = False + model.phase = WorkflowPhase.PAPER_WRITING + elif invariant_id == "outputs.papers_only_skips_proof_work": + model.mode = WorkflowMode.AUTONOMOUS + model.allow_mathematical_proofs = False + model.lean.invocations = 1 + elif invariant_id == "provider.pause_preserves_checkpoint": + model.phase = WorkflowPhase.PAUSED + model.checkpoint = {"paused": True} + elif invariant_id == "provider.stop_resume_preserves_pause": + model.checkpoint = {"paused": True, "stopped": True} + elif invariant_id == "provider.reset_wakes_without_corrupting_checkpoint": + model.provider.pause_count = 1 + model.checkpoint = {"paused": False} + elif invariant_id in {"assistant.no_validator_injection", "prompt.validator_excludes_assistant_memory"}: + if invariant_id.startswith("assistant."): + model.assistant.validator_injections = 1 + else: + model.validator_prompt_has_assistant_memory = True + elif invariant_id == "assistant.disable_clears_live_pack": + model.assistant.enabled = False + model.assistant.live_pack = ("stale-proof",) + elif invariant_id == "assistant.non_blocking_retrieval": + model.assistant.blocked_parent_count = 1 + elif invariant_id == "assistant.stagnant_pack_backoff_not_shutdown": + model.assistant.stagnant_backoff_count = model.assistant.shutdown_count = 1 + elif invariant_id == "proof_scope.manual_not_in_autonomous_current": + model.autonomous_proofs = model.manual_proofs_active = {"same-proof"} + elif invariant_id == "proof_scope.autonomous_events_not_manual": + model.events.append(WorkflowEvent("proof_verified", {"scope": "manual", "proof_id": "auto-proof"})) + elif invariant_id == "proof_scope.manual_clear_archives_active": + model.manual_proofs_before_clear = {"manual-proof"} + elif invariant_id in { + "proof_scope.generated_appendices_stripped_from_prompts", + "prompt.generated_appendices_stripped", + }: + model.generated_appendices_stripped = False + elif invariant_id == "state.clear_removes_active_preserves_history": + model.active_state_cleared = True + model.history_preserved = False + elif invariant_id == "state.completed_brainstorm_no_replay_handoff": + model.completed_brainstorm_generated_paper = model.replayed_completed_brainstorm_handoff = True + elif invariant_id == "state.pruned_papers_excluded_from_context": + model.pruned_papers = model.model_context_papers = {"paper-1"} + elif invariant_id == "state.runtime_roots_are_active_roots": + model.runtime_root_violations = 1 + elif invariant_id == "events.frontend_events_include_scope_phase": + model.events.append(WorkflowEvent("proof_progress", {})) + elif invariant_id == "events.context_overflow_route_identity": + model.events.append(WorkflowEvent("context_overflow_error", {})) + elif invariant_id == "events.context_overflow_persists_across_reload": + model.events.append(WorkflowEvent("context_overflow_error", {"fatal": True})) + elif invariant_id == "events.context_overflow_terminal_stop_once": + model.events.append(WorkflowEvent("context_overflow_error", {"fatal": True})) + elif invariant_id == "events.proof_context_overflow_nonfatal": + model.events.append(WorkflowEvent("proof_context_overflow", {"fatal": True})) + elif invariant_id == "events.proof_verified_after_registration": + model.events.append(WorkflowEvent("proof_verified", {"proof_id": "missing-proof"})) + elif invariant_id == "api.hosted_desktop_only_routes_unavailable": + model.hosted_route_attempts = {"pdf"} + elif invariant_id == "prompt.user_prompt_direct_injected": + model.prompt_user_direct_injected = False + elif invariant_id == "prompt.proof_source_context_required": + model.proof_source_context_present = False + elif invariant_id == "prompt.direct_sources_excluded_from_rag": + model.direct_sources_excluded_from_rag = False + elif invariant_id == "prompt.mandatory_source_overflow_fails_visible": + model.mandatory_source_overflow_visible = False + else: + raise AssertionError(f"No negative fixture for {invariant_id}") + + +def test_first_build_catalog_has_stable_ids_fields_and_model_coverage(): + invariant_ids = [spec.invariant_id for spec in INVARIANT_CATALOG] + + assert len(invariant_ids) == len(set(invariant_ids)) + assert EXPECTED_FIRST_BUILD_FIELDS.issubset(FIRST_BUILD_FIELDS) + assert set(invariant_ids) == set(SCENARIOS_BY_INVARIANT) + assert invariant_ids_for_adapter("model") == set(invariant_ids) + assert set(INVARIANTS_BY_ID) == PLAN2_INVARIANT_IDS + + for spec in INVARIANT_CATALOG: + assert spec.description + assert spec.crossed_fields + + +@pytest.mark.parametrize("spec", INVARIANT_CATALOG, ids=lambda spec: spec.invariant_id) +def test_first_build_catalog_invariants_run_against_model_scenarios(tmp_path, spec): + model = WorkflowModel(runtime_root=tmp_path / spec.invariant_id.replace(".", "_")) + if spec.invariant_id.startswith("proof_scope.manual_"): + model.lean.enabled = True + + run_actions(model, SCENARIOS_BY_INVARIANT[spec.invariant_id]) + + assert_invariant(model, spec.invariant_id) + + +@pytest.mark.parametrize("spec", INVARIANT_CATALOG, ids=lambda spec: spec.invariant_id) +def test_every_catalog_checker_rejects_a_minimal_violation_with_replay(tmp_path, spec): + model = WorkflowModel(runtime_root=tmp_path / "negative") + _make_invariant_violation(model, spec.invariant_id) + + with pytest.raises(AssertionError, match="Replay:") as exc_info: + assert_invariant(model, spec.invariant_id) + + assert "inject_invariant_violation" in str(exc_info.value) diff --git a/tests/workflow_scenarios/test_workflow_scenarios.py b/tests/workflow_scenarios/test_workflow_scenarios.py new file mode 100644 index 0000000..40c9d33 --- /dev/null +++ b/tests/workflow_scenarios/test_workflow_scenarios.py @@ -0,0 +1,92 @@ +from __future__ import annotations + +from tests.workflow_harness.actions import ( + clear, + complete_brainstorm, + complete_topic_exploration, + disable_session_history, + refresh_assistant_pack, + reset_credit, + resume, + run_actions, + run_manual_proof_check, + simulate_credit_exhaustion, + start_autonomous_papers_and_proofs, + start_autonomous_proofs_only, + start_manual_compiler, + stop, +) +from tests.workflow_harness.invariants import assert_all_invariants +from tests.workflow_harness.model import WorkflowModel, WorkflowPhase + + +def test_proofs_only_brainstorm_completion_returns_to_topic_exploration(tmp_path): + model = WorkflowModel(runtime_root=tmp_path) + + run_actions( + model, + [ + start_autonomous_proofs_only, + complete_topic_exploration, + complete_brainstorm, + ], + ) + + assert model.phase is WorkflowPhase.TOPIC_EXPLORATION + assert "auto-proof-1" in model.autonomous_proofs + assert model.lean.invocations == 1 + + +def test_provider_credit_pause_preserves_checkpoint_across_stop_resume(tmp_path): + model = WorkflowModel(runtime_root=tmp_path) + + run_actions( + model, + [ + start_autonomous_papers_and_proofs, + complete_topic_exploration, + simulate_credit_exhaustion, + complete_brainstorm, + stop, + resume, + reset_credit, + ], + ) + + assert model.phase is WorkflowPhase.TIER1_AGGREGATION + assert model.checkpoint.get("paused") is False + assert_all_invariants(model) + + +def test_disabling_session_history_clears_live_assistant_pack(tmp_path): + model = WorkflowModel(runtime_root=tmp_path) + + run_actions( + model, + [ + start_autonomous_papers_and_proofs, + refresh_assistant_pack, + disable_session_history, + ], + ) + + assert model.assistant.live_pack == () + + +def test_manual_clear_archives_active_manual_proofs_without_autonomous_leak(tmp_path): + model = WorkflowModel(runtime_root=tmp_path) + model.lean.enabled = True + + run_actions( + model, + [ + start_manual_compiler, + run_manual_proof_check, + clear, + ], + ) + + assert model.manual_proofs_active == set() + assert model.manual_proofs_archived == {"manual-proof-1"} + assert model.autonomous_proofs == set() +