Part of the Cortex Local Agent documentation.
The repository has five routinely discovered test layers plus one separately invoked integration lane:
- Node tests in
tests\*.test.mjsfor backend/runtime behavior, including guarded PostgreSQL and 2-GiB cases that are skipped until explicitly enabled. - Matbot CLI tests for loader behavior, ephemeral attachments, and workspace-storage isolation.
- Six Playwright WebUI specs in
tests\webui\for browser interactions against the real static WebUI and a fake Matbot server. - The Playwright README question set in
tests\readme-qa\for answer quality against a running Cortex, scored 0/1 by static rules. - Platform-guarded Node cases for Windows PowerShell, local Python sidecars, and the disposable Windows lifecycle.
- A separately invoked disposable Docker/CUDA integration lane in
tests\docker-stack.integration.mjs. - An opt-in OpenRouter live smoke lane in
tests\openrouter-live.integration.mjs. It skips before starting a network request unless an explicitly supplied workspace configuration contains a selected OpenRouter profile with a resolvable key reference and model name.
Run the complete suite before treating a change as verified:
npm run test:allRun only the Node tests:
npm testRun the Matbot CLI isolation tests:
npm run test:cliRun the complete Playwright WebUI suite:
npm run test:webuiRun the quota-consuming OpenRouter smoke only against an explicitly selected
workspace profile (it is intentionally outside test:all):
$env:CORTEX_OPENROUTER_INTEGRATION = '1'
$env:CORTEX_OPENROUTER_CONFIG = (Resolve-Path .\local-agent\matbot\workspaces\private\matbot.yaml).Path
$env:CORTEX_OPENROUTER_PROFILE = 'OpenRouter Chat'
npm run test:integration:openrouterFirst Playwright setup on a machine:
npx playwright install chromiumUseful Playwright variants:
# Desktop WebUI project only
npm run test:webui -- --project chromium
# Mobile WebUI project only
npm run test:webui -- --project mobile-chromium
# Run tests whose title matches a feature area
npm run test:webui -- --grep "workspace RAG"
# Debug a Playwright run locally
npm run test:webui -- --project chromium --headed --debugThe Playwright config starts tests\webui\harness.mjs on
http://127.0.0.1:19787 and serves the same static frontend files used by the
Node WebUI. The harness implements fake Matbot transport endpoints for sessions,
tools, workspaces, files, plugins, skills, remembered facts, experts, and
workspace RAG, traces, evaluation suites, and ROI summaries. It validates WebUI behavior without calling real providers,
spending model tokens, writing production memory stores, or touching live RAG
databases.
Traces are retained on failure. Inspect a failing trace with:
npx playwright show-trace <path-to-trace.zip>Current Playwright coverage includes:
- shell load, providers, conversations, files, plugins, and skills;
- provider and font preference persistence;
- compatible plugin activation/deactivation and incompatible plugin display;
- workspace selector create/rename/delete/switch, including delete cancellation;
- workspace RAG settings save and ingestion progress display;
- remembered facts persisting across conversations;
- remembered facts isolated between workspaces;
- per-test memory reset in the Playwright harness;
- in-page memory browser create/edit/filter/delete/close behavior;
contextual_searchretrieval from remembered facts plus workspace RAG context;- workspace RAG retrieval during conversation;
- source and connector health transport behavior;
- governed SQL planning, approval, execution, citations, and transient-error retry;
- workflow approval inspection and decisions;
- context graph relationship evidence;
- durable expert-review creation, recommendations, checklists, and risks;
- evaluation trace inspection and safe replay, regression-suite execution, sponsor metrics, responsive layout, and stale-selection protection;
- architecture tab semantics and keyboard navigation;
- expert panel all-expert and selected-expert composer flows;
- streaming output, tools, usage, and elapsed-time summary;
- interactive prompt controls;
- workspace file upload/delete;
- skill editor metadata, trigger controls, and confirmed deletion;
- session rename/hide/mark controls;
- send/stop busy behavior;
- mobile sidebar behavior.
tests\readme-qa\readme-questions.json is a static question set derived from
README.md. Every entry carries the question, the README passage that supports
it (sourceQuote), and a rubric of regular expressions:
expect.required— groups of alternatives; each group must match once;expect.forbidden— patterns that must not appear.
An answer scores 1 only when every required group matches and no forbidden
pattern does, otherwise 0. No model grades the answers, so the same answer
always produces the same score and a failure names the rule that was missed.
tests\readme-qa\score.mjs holds the scoring, normalizing markdown backticks
and whitespace before matching.
tests\readme-qa.test.mjs runs inside npm test and validates the rubric
itself without touching a model or the network: ids are unique, all patterns
compile, every sourceQuote still appears in README.md, each quote scores 1
against its own question, and non-answers score 0. A README edit that
invalidates a question fails there first.
The Playwright suite tests\readme-qa\readme-qa.spec.mjs asks each question
through the WebUI composer of a running Cortex — not the fake harness — so
real retrieval and a real provider answer it. Start Cortex first:
scripts\run.ps1
npm run test:readme-qaEach question runs in a fresh conversation, one Playwright test per question, so
the run reports 34 individual pass/fail results. Failures do not stop the suite;
every question is always scored. The run writes
test-results\readme-qa\report.json, report.md (the 0/1 table plus the reason
for each miss), and answers.jsonl (raw scored answers, appended as the run
proceeds so results survive Playwright's worker restart after a failure).
This suite is intentionally excluded from npm run test:all: it needs a live
Cortex and spends provider tokens.
Useful variants:
# Score a single question by id
npm run test:readme-qa -- --grep "workspace-rag-cuda"
# Point at another Cortex instance
$env:CORTEX_WEBUI_URL = "http://127.0.0.1:19778"; npm run test:readme-qa
# Change what the assistant is told to answer from (retrieval probe)
$env:CORTEX_QA_PROMPT_PREFIX = "Answer from the workspace documentation. Question: "; npm run test:readme-qaCORTEX_QA_ANSWER_TIMEOUT_MS (default 180000) and
CORTEX_QA_TEST_TIMEOUT_MS (default 240000) bound a slow turn.
Reading a failing run means separating three causes, which the stored answer tells apart at a glance:
- retrieval miss — the answer says the README does not cover it. The corpus,
not the rubric, is at fault: the workspace RAG context has to include
README.mdbefore the score means anything. - rubric too strict — the answer is right but worded differently. Add the alternative to that required group.
- wrong answer — the content is confidently incorrect. This is the failure the suite exists to catch; do not widen the rubric to make it pass.
Tuning a pattern does not need another provider run. rescore.mjs re-applies
the current rubric to the answers of the last run and names every question whose
score moved:
node tests/readme-qa/rescore.mjs
node tests/readme-qa/rescore.mjs --verbose # print the stored answer for each miss
node tests/readme-qa/rescore.mjs --write # rewrite report.json / report.mdAdding a question means adding one JSON entry — question, sourceQuote copied
from README.md, and the required/forbidden patterns. npm test then proves
the entry is answerable from the document before any model sees it.
Node tests cover:
- production memory capture using the real cognition tool and filesystem store;
- memory recall between conversations: a fact stated in one conversation reaching
the next, sharing across every conversation in a workspace, and staying inside
that workspace (
tests\memory-recall.test.mjs); - memory persistence after runtime recreation and isolation across workspace roots;
- production
dream_timefact processing and run-record isolation; - production memory-browser workspace switching and CAS conflict handling;
- before/after production-memory hashes to prevent test pollution;
- file-index storage and search;
- file-broker policy and write backups;
- hybrid KnowledgeIndex ranking/deduplication;
- expert-panel plugin behavior and isolated retrieval;
- workspace-rag runtime ingestion flow;
- end-to-end span capture, redaction, safe replay, deterministic and model-based scorers, model-cost accounting, workflow outcome linkage, and ROI arithmetic;
- harness tool-use runtime behavior — schema validation, loop budgets, parallel
execution, doom-loop interception, orphaned tool-call finalization, output
truncation, and permission gate flows (
tests\tool-use-runtime.test.mjs, spec: Tool Use Specification); - the harness tools
read/write/edit/glob/grep/list/todowriteend-to-end against temp-directory workspaces, including workspace confinement, read-before-edit enforcement, CRLF preservation, and result caps (tests\tool-use-harness.test.mjs); - provider adapter error fidelity:
is_errorserialization on the OpenAI-compatible wire and degraded handling of truncated tool-call arguments (tests\tool-use-adapters.test.mjs).
Run only the evaluation and ROI backend coverage:
node --test tests/evaluation-observability.test.mjsRun only the cross-conversation memory coverage:
node --test tests/memory-recall.test.mjsRun only the workspace-switch handoff coverage:
node --test tests/workspace-switch.test.mjsThe ordinary suite never starts Docker, rebuilds CUDA images, downloads an
embedding model, or deletes volumes. The guarded integration command creates a
generated cortex-test-* Compose project, an isolated temporary environment
file, and project-owned volumes; its cleanup removes only that generated
project.
Run it only on a disposable Docker host:
$env:CORTEX_DOCKER_INTEGRATION = "1"
npm run test:integration:dockerThe test checks Postgres/pgvector, Neo4j authentication, Mem0 API reachability,
and the documented password-rotation behaviour. Rotation intentionally uses
down --volumes, so the test proves fixture data is gone after recreation; it
does not and must not target a developer's existing Mem0 data.
CUDA checks require both Docker opt-in and an available NVIDIA Docker runtime:
$env:CORTEX_DOCKER_INTEGRATION = "1"
$env:CORTEX_CUDA_INTEGRATION = "1"
npm run test:integration:cudaThis starts the MiniLM CUDA sidecar and polls its real health payload. To also
exercise the larger E5 model download, set CORTEX_CUDA_E5_INTEGRATION=1; it is
separate because a cold model cache materially increases run time.
The deterministic CUDA contract tests remain in npm test: they validate the
model, embedding dimensions, profile, normalization, and E5 prefixes against a
local fake sidecar without requiring a GPU.
That suite covers the two properties a switch depends on: shutdown releasing the web port promptly even with a connection stuck mid-request, and the workspace listing identifying the process that answered it — without which the WebUI cannot tell a completed switch from the outgoing process still serving the previous workspace's conversations.
Each recall scenario is reported as its own named subtest, so a regression names
the behavior that broke. The scenarios drive the real capture tool, recall hook,
and contextual_search over temp-directory stores — only the model is faked —
and the suite hashes every workspace's remembered_facts before and after to
prove it never touched real memory.
Inventory snapshot: 2026-09-05. It contains 535 named test cases or named Node subtests across 94 test source files, including 24 C1–C14 migration cases. A Playwright definition is listed once even when the desktop and mobile projects both discover it. Generated README questions and the named memory/workspace-switch subtests are expanded because those are the titles the runners report.
The catalog describes source-defined behavior, not a coverage-percentage claim. "Excludes" records what a passing test does not prove. "Assumptions" records the environment and fidelity boundary on which the assertion relies.
| Lane | Command | Cataloged titles | Discovery boundary |
|---|---|---|---|
| Default Node test lane | npm test |
358 | tests/*.test.mjs; guarded PostgreSQL and 2-GiB definitions are discovered but skipped without opt-in. |
| Matbot CLI test lane | npm run test:cli |
5 | Three CLI test files through the Matbot TypeScript loader. |
| Harness-backed Playwright WebUI lane | npm run test:webui |
135 | Six specs against the fake harness, including a bundled browser runtime fixture; definitions may be discovered by both Chromium projects and then project-skipped. |
| Live README question lane | npm run test:readme-qa |
34 | 34 generated cases against a separately running real Cortex; excluded from test:all. |
| Guarded Docker and CUDA lane | npm run test:integration:docker / npm run test:integration:cuda |
3 | One guarded integration file; excluded from test:all. |
npm run test:all is the composition npm test && npm run test:cli && npm run test:webui. It does not run the live README QA or disposable Docker/CUDA commands. eval:cortex, verify:openai, typechecking, builds, and documentation checks are useful verification commands but are not test-runner definitions, so they are not represented as test titles below.
| Profile | Excludes | Assumptions |
|---|---|---|
| C - Matbot CLI | WebUI behavior, live providers, and external services; stores and modules are isolated fixtures. | Node >=24, pnpm, the Matbot TypeScript loader, and writable temporary storage. |
| D - disposable Docker/CUDA | test:all, existing developer Compose projects/volumes, live providers, and browser behavior. |
Explicit Docker opt-in on a disposable host; CUDA cases also need NVIDIA runtime and their dedicated opt-ins. |
| G - guarded 2-GiB parser | Default enabled execution and dense real-world 2-GiB content; the fixture is sparse. | CORTEX_RAG_V2_2GB_INTEGRATION=1, sparse-file support, enough address space, disk quota, and time. |
| L - disposable Windows lifecycle | Default enabled execution and non-Windows hosts; may start/stop services and mutate then restore test secrets. | All documented lifecycle opt-ins, a dedicated disposable Windows host, Docker/services, and a fake provider for the provider journey. |
| N - deterministic Node | Live model providers, a real browser, Docker/GPU, and production workspace data. | Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| P - guarded PostgreSQL/pgvector | Default enabled execution, non-PostgreSQL backends, live providers, and browser behavior. | CORTEX_RAG_V2_POSTGRES_INTEGRATION=1, reachable pgvector PostgreSQL, disposable schemas, and sufficient role rights for RLS coverage. |
| Q - live README QA | test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
| S - static source/document contract | Runtime execution of the documented feature and behavior outside the inspected files or literal contract. | The checked-in documents, configuration, paths, labels, and source literals are the authoritative shipped contract. |
| W - Playwright harness | Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
| Y - local Python sidecar | A real GPU, downloaded production model guarantees, live providers, and remote sidecars. | A usable Python interpreter and the sidecar's local dependencies; the test is skipped when Python is absent. |
| Z - Windows PowerShell | Non-Windows execution and non-PowerShell shells; no production scripts or secrets are used. | Windows PowerShell is available and temporary scripts/processes can be created and terminated. |
Runner: npm test. Titles in this section: 277.
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
REL-04 real missing executable settles without crashing the host |
Exercises a real missing executable and controlled child/pipe errors and cancellation. | Parent survival, terminal occurrence states, idempotent settlement, and bounded termination/draining. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-04 spawn and input-pipe errors settle once and tolerate trailing exit |
Exercises a real missing executable and controlled child/pipe errors and cancellation. | Parent survival, terminal occurrence states, idempotent settlement, and bounded termination/draining. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-04 cancellation and output draining are bounded |
Exercises a real missing executable and controlled child/pipe errors and cancellation. | Parent survival, terminal occurrence states, idempotent settlement, and bounded termination/draining. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-04 scheduler persists launch failure as a terminal occurrence |
Exercises a real missing executable and controlled child/pipe errors and cancellation. | Parent survival, terminal occurrence states, idempotent settlement, and bounded termination/draining. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-09 scheduler recovers from read failure without duplicating an occurrence |
Injects schedule read/write failures and lost acknowledgements around launch, completion, and recovery. | Automatic retry, at-most-one local launch, preserved control state, resume rearming, and deferred uncertain work. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-09 scheduler recovers from prepare failure without duplicating an occurrence |
Injects schedule read/write failures and lost acknowledgements around launch, completion, and recovery. | Automatic retry, at-most-one local launch, preserved control state, resume rearming, and deferred uncertain work. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-09 scheduler recovers from prepare-ack failure without duplicating an occurrence |
Injects schedule read/write failures and lost acknowledgements around launch, completion, and recovery. | Automatic retry, at-most-one local launch, preserved control state, resume rearming, and deferred uncertain work. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-09 scheduler recovers from complete failure without duplicating an occurrence |
Injects schedule read/write failures and lost acknowledgements around launch, completion, and recovery. | Automatic retry, at-most-one local launch, preserved control state, resume rearming, and deferred uncertain work. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-09 scheduler recovers from complete-ack failure without duplicating an occurrence |
Injects schedule read/write failures and lost acknowledgements around launch, completion, and recovery. | Automatic retry, at-most-one local launch, preserved control state, resume rearming, and deferred uncertain work. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-09 suspension survives completion persistence retries and resume rearms a stopped record |
Injects schedule read/write failures and lost acknowledgements around launch, completion, and recovery. | Automatic retry, at-most-one local launch, preserved control state, resume rearming, and deferred uncertain work. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-09 restart records an uncertain running occurrence without replaying it |
Injects schedule read/write failures and lost acknowledgements around launch, completion, and recovery. | Automatic retry, at-most-one local launch, preserved control state, resume rearming, and deferred uncertain work. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-09 a lost acknowledgement while deferring uncertain work still honours the new due time |
Injects schedule read/write failures and lost acknowledgements around launch, completion, and recovery. | Automatic retry, at-most-one local launch, preserved control state, resume rearming, and deferred uncertain work. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
REL-07 cancellation interrupts status backoff without another request |
Injects retryable network/status failures with a cancelled backoff. | Prompt cancellation, no extra requests, and removal of retry wait listeners. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-07 cancellation interrupts network backoff without another request |
Injects retryable network/status failures with a cancelled backoff. | Prompt cancellation, no extra requests, and removal of retry wait listeners. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-07 pre-cancellation sends nothing and successful retry waits remove listeners |
Injects retryable network/status failures with a cancelled backoff. | Prompt cancellation, no extra requests, and removal of retry wait listeners. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-06 OpenAICompatAdapter bounds headers, idle streams, and total completion / headers |
Streams controlled responses from a loopback HTTP server, including stalls and heartbeats. | Header, idle, and total deadlines; caller cancellation; setting validation; and healthy long streams. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-06 OpenAICompatAdapter bounds headers, idle streams, and total completion / idle |
Streams controlled responses from a loopback HTTP server, including stalls and heartbeats. | Header, idle, and total deadlines; caller cancellation; setting validation; and healthy long streams. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-06 OpenAICompatAdapter bounds headers, idle streams, and total completion / heartbeat |
Streams controlled responses from a loopback HTTP server, including stalls and heartbeats. | Header, idle, and total deadlines; caller cancellation; setting validation; and healthy long streams. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-06 OpenAICompatAdapter bounds headers, idle streams, and total completion / total |
Streams controlled responses from a loopback HTTP server, including stalls and heartbeats. | Header, idle, and total deadlines; caller cancellation; setting validation; and healthy long streams. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-06 OpenAICompatAdapter bounds headers, idle streams, and total completion |
Streams controlled responses from a loopback HTTP server, including stalls and heartbeats. | Header, idle, and total deadlines; caller cancellation; setting validation; and healthy long streams. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-06 OpenAICompatAdapter healthy streaming outlives the header attempt timer |
Streams controlled responses from a loopback HTTP server, including stalls and heartbeats. | Header, idle, and total deadlines; caller cancellation; setting validation; and healthy long streams. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-06 OpenAICompatAdapter respects cancellation and validates deadline settings |
Streams controlled responses from a loopback HTTP server, including stalls and heartbeats. | Header, idle, and total deadlines; caller cancellation; setting validation; and healthy long streams. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-06 AnthropicAdapter bounds headers, idle streams, and total completion / headers |
Streams controlled responses from a loopback HTTP server, including stalls and heartbeats. | Header, idle, and total deadlines; caller cancellation; setting validation; and healthy long streams. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-06 AnthropicAdapter bounds headers, idle streams, and total completion / idle |
Streams controlled responses from a loopback HTTP server, including stalls and heartbeats. | Header, idle, and total deadlines; caller cancellation; setting validation; and healthy long streams. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-06 AnthropicAdapter bounds headers, idle streams, and total completion / heartbeat |
Streams controlled responses from a loopback HTTP server, including stalls and heartbeats. | Header, idle, and total deadlines; caller cancellation; setting validation; and healthy long streams. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-06 AnthropicAdapter bounds headers, idle streams, and total completion / total |
Streams controlled responses from a loopback HTTP server, including stalls and heartbeats. | Header, idle, and total deadlines; caller cancellation; setting validation; and healthy long streams. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-06 AnthropicAdapter bounds headers, idle streams, and total completion |
Streams controlled responses from a loopback HTTP server, including stalls and heartbeats. | Header, idle, and total deadlines; caller cancellation; setting validation; and healthy long streams. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-06 AnthropicAdapter healthy streaming outlives the header attempt timer |
Streams controlled responses from a loopback HTTP server, including stalls and heartbeats. | Header, idle, and total deadlines; caller cancellation; setting validation; and healthy long streams. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-06 AnthropicAdapter respects cancellation and validates deadline settings |
Streams controlled responses from a loopback HTTP server, including stalls and heartbeats. | Header, idle, and total deadlines; caller cancellation; setting validation; and healthy long streams. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-08 error before content fails without releasing tool calls or a done event |
Feeds provider errors, missing terminals, trailing usage, and unconfirmed tool streams. | Explicit failure, preserved partial history, withheld unconfirmed calls, and one successful terminal event. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-08 error after content fails without releasing tool calls or a done event |
Feeds provider errors, missing terminals, trailing usage, and unconfirmed tool streams. | Explicit failure, preserved partial history, withheld unconfirmed calls, and one successful terminal event. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-08 EOF after content fails without releasing tool calls or a done event |
Feeds provider errors, missing terminals, trailing usage, and unconfirmed tool streams. | Explicit failure, preserved partial history, withheld unconfirmed calls, and one successful terminal event. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-08 unconfirmed tool fails without releasing tool calls or a done event |
Feeds provider errors, missing terminals, trailing usage, and unconfirmed tool streams. | Explicit failure, preserved partial history, withheld unconfirmed calls, and one successful terminal event. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-08 error after tool finish fails without releasing tool calls or a done event |
Feeds provider errors, missing terminals, trailing usage, and unconfirmed tool streams. | Explicit failure, preserved partial history, withheld unconfirmed calls, and one successful terminal event. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-08 finish plus trailing usage produces one terminal event and ends at DONE |
Feeds provider errors, missing terminals, trailing usage, and unconfirmed tool streams. | Explicit failure, preserved partial history, withheld unconfirmed calls, and one successful terminal event. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
E2E-020 / T2-E2E-006 / MISSING-10 background scheduling is persistent, manageable, principal-bound, and workspace-isolated |
Creates state, crosses the stated persistence or restart boundary, and checks this outcome: background scheduling is persistent, manageable, principal-bound, and workspace-isolated. | governed workflows, approvals, scheduling, or structured-data execution; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
H4: broker rejects read/write through links that escape or are themselves reparse points |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: broker rejects read/write through links that escape or are themselves reparse points. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
H4: openVerified rejects a final-component link resolving inside the workspace |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: openVerified rejects a final-component link resolving inside the workspace. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
L13: backup retention prunes oldest-first beyond CORTEX_FILE_BROKER_MAX_BACKUPS |
Runs the focused fixture against the implementation and checks this outcome: backup retention prunes oldest-first beyond CORTEX_FILE_BROKER_MAX_BACKUPS. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
L14: a mid-file insertion renders as one small hunk, not whole-tail replacement |
Runs the focused fixture against the implementation and checks this outcome: a mid-file insertion renders as one small hunk, not whole-tail replacement. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
L14: edits and deletions produce correctly counted unified hunks |
Runs the focused fixture against the implementation and checks this outcome: edits and deletions produce correctly counted unified hunks. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
CTA-4/5/6/7 dream run inspection and cognition configuration use the direct tools |
Runs the focused fixture against the implementation and checks this outcome: 5/6/7 dream run inspection and cognition configuration use the direct tools. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
CTA-8 cognition blocklist excludes matching skills from dream-time routing |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: cognition blocklist excludes matching skills from dream-time routing. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
M3: '#' inside double-quoted values survives comment stripping |
Creates state, crosses the stated persistence or restart boundary, and checks this outcome: '#' inside double-quoted values survives comment stripping. | configuration parsing, validation, compatibility, and model/provider normalization; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
M3: '#' inside single-quoted values survives comment stripping |
Creates state, crosses the stated persistence or restart boundary, and checks this outcome: '#' inside single-quoted values survives comment stripping. | configuration parsing, validation, compatibility, and model/provider normalization; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
M3: real comments after quoted values are still stripped |
Runs the focused fixture against the implementation and checks this outcome: real comments after quoted values are still stripped. | configuration parsing, validation, compatibility, and model/provider normalization; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
M4a: sequence items containing mappings parse as records |
Runs the focused fixture against the implementation and checks this outcome: sequence items containing mappings parse as records. | configuration parsing, validation, compatibility, and model/provider normalization; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
M4a: plain scalars containing ':' without ': ' stay scalars |
Runs the focused fixture against the implementation and checks this outcome: plain scalars containing ':' without ': ' stay scalars. | configuration parsing, validation, compatibility, and model/provider normalization; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
M4a: sequences of mappings flow through parseConfig's available_models |
Runs the focused fixture against the implementation and checks this outcome: sequences of mappings flow through parseConfig's available_models. | configuration parsing, validation, compatibility, and model/provider normalization; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
M4b: literal block scalar defaults to clip chomping |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: literal block scalar defaults to clip chomping. | configuration parsing, validation, compatibility, and model/provider normalization; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
M4b: '|-' strips the trailing newline |
Runs the focused fixture against the implementation and checks this outcome: '|-' strips the trailing newline. | configuration parsing, validation, compatibility, and model/provider normalization; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
M4b: '|+' keeps the trailing newline like clip when no blank lines follow |
Runs the focused fixture against the implementation and checks this outcome: '|+' keeps the trailing newline like clip when no blank lines follow. | configuration parsing, validation, compatibility, and model/provider normalization; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
M4b: '>' folds lines with spaces (clip) |
Runs the focused fixture against the implementation and checks this outcome: '>' folds lines with spaces (clip). | configuration parsing, validation, compatibility, and model/provider normalization; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
M4b: '>-' strips the folded result |
Runs the focused fixture against the implementation and checks this outcome: '>-' strips the folded result. | configuration parsing, validation, compatibility, and model/provider normalization; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
L5: duplicate model names inside one group warn instead of silently overwriting |
Runs the focused fixture against the implementation and checks this outcome: duplicate model names inside one group warn instead of silently overwriting. | configuration parsing, validation, compatibility, and model/provider normalization; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
L5: openai_compatible group colliding with a providers key warns |
Runs the focused fixture against the implementation and checks this outcome: openai_compatible group colliding with a providers key warns. | configuration parsing, validation, compatibility, and model/provider normalization; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
L5: object-valued parameters are warned about and skipped |
Runs the focused fixture against the implementation and checks this outcome: object-valued parameters are warned about and skipped. | configuration parsing, validation, compatibility, and model/provider normalization; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
L5: numeric strings for known numeric parameters are coerced |
Runs the focused fixture against the implementation and checks this outcome: numeric strings for known numeric parameters are coerced. | configuration parsing, validation, compatibility, and model/provider normalization; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T3-E2E-027 connector-fabric runtime flow passes under the Matbot TypeScript loader |
Runs the focused fixture against the implementation and checks this outcome: connector-fabric runtime flow passes under the Matbot TypeScript loader. | architecture sources, connectors, graph evidence, provenance, or health; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T3-E2E-026 context-graph runtime flow covers multi-hop provenance and confidence bounds |
Runs the focused fixture against the implementation and checks this outcome: context-graph runtime flow covers multi-hop provenance and confidence bounds. | architecture sources, connectors, graph evidence, provenance, or health; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
knowledge index search skips empty terms instead of hanging |
Runs the focused fixture against the implementation and checks this outcome: knowledge index search skips empty terms instead of hanging. | core hardening behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
session is persisted when a toolcall hook aborts the turn |
Creates state, crosses the stated persistence or restart boundary, and checks this outcome: session is persisted when a toolcall hook aborts the turn. | core hardening behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
pump prelude failures surface as an error event, not an unhandled rejection |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: pump prelude failures surface as an error event, not an unhandled rejection. | core hardening behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
teardownPlugins attributes errors to the plugin that failed |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: teardownPlugins attributes errors to the plugin that failed. | core hardening behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
unloadPlugin clears its teardown timeout timer |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: unloadPlugin clears its teardown timeout timer. | core hardening behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
settings CAS uses unique versions and bounds retries under contention |
Runs the focused fixture against the implementation and checks this outcome: settings CAS uses unique versions and bounds retries under contention. | core hardening behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
observability spans record scrubbed tool input and result |
Runs the focused fixture against the implementation and checks this outcome: observability spans record scrubbed tool input and result. | core hardening behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
system-context build isolates a throwing contributor |
Creates separate security or workspace scopes and checks this outcome: system-context build isolates a throwing contributor. | core hardening behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
vault scrub redacts overlapping secrets fully regardless of insertion order |
Runs the focused fixture against the implementation and checks this outcome: vault scrub redacts overlapping secrets fully regardless of insertion order. | core hardening behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
parseSSE accepts spec-compliant data lines without a space |
Runs the focused fixture against the implementation and checks this outcome: parseSSE accepts spec-compliant data lines without a space. | core hardening behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
malformed apiVersion warns instead of silently skipping the version check |
Runs the focused fixture against the implementation and checks this outcome: malformed apiVersion warns instead of silently skipping the version check. | core hardening behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
tool collision resolution overwrites only on explicit affirmative answers |
Runs the focused fixture against the implementation and checks this outcome: tool collision resolution overwrites only on explicit affirmative answers. | core hardening behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
knowledge index dedups by id and exposes docs as a read-only snapshot |
Runs the focused fixture against the implementation and checks this outcome: knowledge index dedups by id and exposes docs as a read-only snapshot. | core hardening behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
fetchWithRetry aborts a hung connection once timeoutMs elapses |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: fetchWithRetry aborts a hung connection once timeoutMs elapses. | core hardening behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
fetchWithRetry respects the overall timeoutMs budget across status retries |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: fetchWithRetry respects the overall timeoutMs budget across status retries. | core hardening behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
fetchWithRetry succeeds within the budget when the first attempt responds |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: fetchWithRetry succeeds within the budget when the first attempt responds. | core hardening behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
applySort compares booleans numerically, numbers numerically, missing last |
Runs the focused fixture against the implementation and checks this outcome: applySort compares booleans numerically, numbers numerically, missing last. | core hardening behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
REL-01 provider failure preserves 0 completed tool rounds and partial output |
Injects provider or session-save failure after completed actions. | Durable paired tool history, preserved partial prose, explicit persistence errors, and suppression of subsequent requests. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-01 provider failure preserves 1 completed tool rounds and partial output |
Injects provider or session-save failure after completed actions. | Durable paired tool history, preserved partial prose, explicit persistence errors, and suppression of subsequent requests. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-01 provider failure preserves 2 completed tool rounds and partial output |
Injects provider or session-save failure after completed actions. | Durable paired tool history, preserved partial prose, explicit persistence errors, and suppression of subsequent requests. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-01 persistence failure is explicit and prevents the next provider request |
Injects provider or session-save failure after completed actions. | Durable paired tool history, preserved partial prose, explicit persistence errors, and suppression of subsequent requests. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-08 iterator exhaustion is a failed provider completion and preserves partial prose |
Feeds provider errors, missing terminals, trailing usage, and unconfirmed tool streams. | Explicit failure, preserved partial history, withheld unconfirmed calls, and one successful terminal event. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T3-E2E-037 dream-time merges a bounded fact cluster, records provenance, and keeps workspace stores isolated |
Creates separate security or workspace scopes and checks this outcome: dream-time merges a bounded fact cluster, records provenance, and keeps workspace stores isolated. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-038 dream-time quarantines a durable merge failure without overwriting the skill |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: dream-time quarantines a durable merge failure without overwriting the skill. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T2-E2E-016 evaluation CLI returns release-gate exit codes and valid escaped JUnit |
Runs the focused fixture against the implementation and checks this outcome: evaluation CLI returns release-gate exit codes and valid escaped JUnit. | evaluation scoring, observability, replay, pricing, release gates, or ROI; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-007 evaluation CLI redacts credentials from stdout and JUnit |
Runs the focused fixture against the implementation and checks this outcome: evaluation CLI redacts credentials from stdout and JUnit. | evaluation scoring, observability, replay, pricing, release gates, or ROI; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T3-E2E-028 evaluation-observability runtime flow covers safe replay, release gates, and ROI |
Runs the focused fixture against the implementation and checks this outcome: evaluation-observability runtime flow covers safe replay, release gates, and ROI. | evaluation scoring, observability, replay, pricing, release gates, or ROI; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T3-E2E-029 model pricing honours direct cost, catalog pricing, cached input, and invalid configuration |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: model pricing honours direct cost, catalog pricing, cached input, and invalid configuration. | evaluation scoring, observability, replay, pricing, release gates, or ROI; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
minimal env exposes PATH-style keys but not unrelated process secrets |
Runs the focused fixture against the implementation and checks this outcome: minimal env exposes PATH-style keys but not unrelated process secrets. | process execution, environment minimization, cwd confinement, timeout, and shell boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
cwd confinement allows paths inside the base directory |
Runs the focused fixture against the implementation and checks this outcome: cwd confinement allows paths inside the base directory. | process execution, environment minimization, cwd confinement, timeout, and shell boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
cwd confinement rejects escapes, absolute foreign paths, and sibling-prefix tricks |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: cwd confinement rejects escapes, absolute foreign paths, and sibling-prefix tricks. | process execution, environment minimization, cwd confinement, timeout, and shell boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
powershell tool passes a canary-free environment with explicit vars through |
Executes the PowerShell boundary on Windows and checks this outcome: powershell tool passes a canary-free environment with explicit vars through. | process execution, environment minimization, cwd confinement, timeout, and shell boundaries; specifically the outcome and boundary stated in this title. | Z: Non-Windows execution and non-PowerShell shells; no production scripts or secrets are used. | Z: Windows PowerShell is available and temporary scripts/processes can be created and terminated. |
powershell tool honors cwd inside the workspace and refuses escapes |
Executes the PowerShell boundary on Windows and checks this outcome: powershell tool honors cwd inside the workspace and refuses escapes. | process execution, environment minimization, cwd confinement, timeout, and shell boundaries; specifically the outcome and boundary stated in this title. | Z: Non-Windows execution and non-PowerShell shells; no production scripts or secrets are used. | Z: Windows PowerShell is available and temporary scripts/processes can be created and terminated. |
bash_config validates input shapes with descriptive errors and ignores unknown fields |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: bash_config validates input shapes with descriptive errors and ignores unknown fields. | process execution, environment minimization, cwd confinement, timeout, and shell boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
container provision lock serializes per name and survives failures |
Creates state, crosses the stated persistence or restart boundary, and checks this outcome: container provision lock serializes per name and survives failures. | process execution, environment minimization, cwd confinement, timeout, and shell boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T3-E2E-025 expert configuration rejects duplicate ids, missing roots, and malformed schema |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: expert configuration rejects duplicate ids, missing roots, and malformed schema. | expert configuration, knowledge isolation, provider choice, synthesis, or durable review; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T3-E2E-035 custom expert configuration isolates roots and preserves all three panel modes |
Creates separate security or workspace scopes and checks this outcome: custom expert configuration isolates roots and preserves all three panel modes. | expert configuration, knowledge isolation, provider choice, synthesis, or durable review; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T3-E2E-023 expert knowledge loads every supported text format and ignores unsupported or oversized files |
Runs the focused fixture against the implementation and checks this outcome: expert knowledge loads every supported text format and ignores unsupported or oversized files. | expert configuration, knowledge isolation, provider choice, synthesis, or durable review; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-024 expert knowledge keeps roots isolated and does not let a large matching file leak into results |
Creates separate security or workspace scopes and checks this outcome: expert knowledge keeps roots isolated and does not let a large matching file leak into results. | expert configuration, knowledge isolation, provider choice, synthesis, or durable review; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
EKI-6 expert panel response warns when a configured knowledge root becomes inaccessible |
Runs the focused fixture against the implementation and checks this outcome: expert panel response warns when a configured knowledge root becomes inaccessible. | expert configuration, knowledge isolation, provider choice, synthesis, or durable review; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T3-E2E-022 EPF provider resolution prefers expert pins and falls back through turn and panel defaults |
Runs the focused fixture against the implementation and checks this outcome: ePF provider resolution prefers expert pins and falls back through turn and panel defaults. | expert configuration, knowledge isolation, provider choice, synthesis, or durable review; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T3-E2E-041 expert reviews persist structured consensus, risks, and links to durable external records |
Creates state, crosses the stated persistence or restart boundary, and checks this outcome: expert reviews persist structured consensus, risks, and links to durable external records. | expert configuration, knowledge isolation, provider choice, synthesis, or durable review; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
expert panel configuration does not override the UI-selected provider |
Reads the shipped source, configuration, or documentation and checks this contract: expert panel configuration does not override the UI-selected provider. | expert configuration, knowledge isolation, provider choice, synthesis, or durable review; specifically the outcome and boundary stated in this title. | S: Runtime execution of the documented feature and behavior outside the inspected files or literal contract. | S: The checked-in documents, configuration, paths, labels, and source literals are the authoritative shipped contract. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
expert panel runs selected experts with isolated knowledge and synthesis |
Creates separate security or workspace scopes and checks this outcome: expert panel runs selected experts with isolated knowledge and synthesis. | expert configuration, knowledge isolation, provider choice, synthesis, or durable review; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
expert panel reports unknown experts as tool errors |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: expert panel reports unknown experts as tool errors. | expert configuration, knowledge isolation, provider choice, synthesis, or durable review; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T3-E2E-004 / MISSING-06 high-risk file-broker writes require approval and cannot escape through junctions |
Runs the focused fixture against the implementation and checks this outcome: high-risk file-broker writes require approval and cannot escape through junctions. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
file-index recognizes file-level API-key and private-key shapes without rejecting ordinary prose |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: file-index recognizes file-level API-key and private-key shapes without rejecting ordinary prose. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
file-index recognizes AWS, GitHub, and Google credential shapes without false positives on lookalikes |
Runs the focused fixture against the implementation and checks this outcome: file-index recognizes AWS, GitHub, and Google credential shapes without false positives on lookalikes. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
file-index redacts credential-shaped assignments across common formats and preserves safe lookalikes |
Runs the focused fixture against the implementation and checks this outcome: file-index redacts credential-shaped assignments across common formats and preserves safe lookalikes. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-036 file-index skips high-confidence secret files and keeps surrounding redacted documents searchable |
Runs the focused fixture against the implementation and checks this outcome: file-index skips high-confidence secret files and keeps surrounding redacted documents searchable. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
filesystem store persists record ids that are not file-name safe |
Creates state, crosses the stated persistence or restart boundary, and checks this outcome: filesystem store persists record ids that are not file-name safe. | filesystem/SQLite storage, CAS, concurrency, migration, watch, and size boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
json-validation rejects over-long regex patterns and fails closed on invalid ones |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: json-validation rejects over-long regex patterns and fails closed on invalid ones. | final hardening behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
WebCryptoVault encrypt/decrypt roundtrips large payloads via chunked base64 |
Creates state, crosses the stated persistence or restart boundary, and checks this outcome: webCryptoVault encrypt/decrypt roundtrips large payloads via chunked base64. | final hardening behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
postgres quoteIdentifier enforces the safe identifier charset |
Runs the focused fixture against the implementation and checks this outcome: postgres quoteIdentifier enforces the safe identifier charset. | final hardening behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T2-E2E-001 documented WebUI boundary is loopback-only |
Runs the focused fixture against the implementation and checks this outcome: documented WebUI boundary is loopback-only. | loopback HTTP service security, protocol, limits, shutdown, and error handling; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
high-cardinality storage migrates source and graph records to SQLite while preserving ordinary filesystem stores |
Runs the focused fixture against the implementation and checks this outcome: high-cardinality storage migrates source and graph records to SQLite while preserving ordinary filesystem stores. | filesystem/SQLite storage, CAS, concurrency, migration, watch, and size boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T2-E2E-021 shared HTTP JSON helpers validate content type, shape, and body size |
Runs the focused fixture against the implementation and checks this outcome: shared HTTP JSON helpers validate content type, shape, and body size. | loopback HTTP service security, protocol, limits, shutdown, and error handling; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
hybrid retrieval migration plan has an executable completed-activity contract |
Reads the shipped source, configuration, or documentation and checks this contract: hybrid retrieval migration plan has an executable completed-activity contract. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | S: Runtime execution of the documented feature and behavior outside the inspected files or literal contract. | S: The checked-in documents, configuration, paths, labels, and source literals are the authoritative shipped contract. |
hybrid retrieval implementation evidence files exist |
Reads the shipped source, configuration, or documentation and checks this contract: hybrid retrieval implementation evidence files exist. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | S: Runtime execution of the documented feature and behavior outside the inspected files or literal contract. | S: The checked-in documents, configuration, paths, labels, and source literals are the authoritative shipped contract. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
E2E-008 / T2-E2E-020 complete Windows first-run configures hidden secrets and starts Docker-backed services |
Runs the opt-in Windows lifecycle journey and checks this outcome: complete Windows first-run configures hidden secrets and starts Docker-backed services. | workspace/process lifecycle, switch fencing, cleanup, and isolation; specifically the outcome and boundary stated in this title. | L: Default enabled execution and non-Windows hosts; may start/stop services and mutate then restore test secrets. | L: All documented lifecycle opt-ins, a dedicated disposable Windows host, Docker/services, and a fake provider for the provider journey. |
T3-E2E-021 disposable Windows lifecycle completes a provider turn and preserves its session across restart |
Runs the opt-in Windows lifecycle journey and checks this outcome: disposable Windows lifecycle completes a provider turn and preserves its session across restart. | workspace/process lifecycle, switch fencing, cleanup, and isolation; specifically the outcome and boundary stated in this title. | L: Default enabled execution and non-Windows hosts; may start/stop services and mutate then restore test secrets. | L: All documented lifecycle opt-ins, a dedicated disposable Windows host, Docker/services, and a fake provider for the provider journey. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T2-E2E-006 file index stores searchable text with path metadata |
Runs the focused fixture against the implementation and checks this outcome: file index stores searchable text with path metadata. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T2-E2E-007 file index rejects index roots outside the configured workspace roots |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: file index rejects index roots outside the configured workspace roots. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T2-E2E-008 file index applies the broker security policy to every indexed file |
Runs the focused fixture against the implementation and checks this outcome: file index applies the broker security policy to every indexed file. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T2-E2E-009 file index exclusion patterns match Windows-style paths and prune whole directories |
Runs the focused fixture against the implementation and checks this outcome: file index exclusion patterns match Windows-style paths and prune whole directories. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T2-E2E-011 file index redacts credential values instead of discarding the whole file |
Runs the focused fixture against the implementation and checks this outcome: file index redacts credential values instead of discarding the whole file. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T2-E2E-012 file index survives an unreadable directory instead of failing the whole run |
Creates state, crosses the stated persistence or restart boundary, and checks this outcome: file index survives an unreadable directory instead of failing the whole run. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T2-E2E-013 workspace config honours the pre-rename excludedPatterns key |
Runs the focused fixture against the implementation and checks this outcome: workspace config honours the pre-rename excludedPatterns key. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T2-E2E-014 file broker blocks writes outside configured roots and allows project writes |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: file broker blocks writes outside configured roots and allows project writes. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T2-E2E-015 file broker config cache reuses unchanged values and reloads changed files |
Runs the focused fixture against the implementation and checks this outcome: file broker config cache reuses unchanged values and reloads changed files. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T2-E2E-016 file broker Matbot tool reads host files through the broker service |
Runs the focused fixture against the implementation and checks this outcome: file broker Matbot tool reads host files through the broker service. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T2-E2E-017 file writer creates a backup and a diff for overwrites |
Runs the focused fixture against the implementation and checks this outcome: file writer creates a backup and a diff for overwrites. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T2-E2E-018 hybrid ranking deduplicates entries |
Runs the focused fixture against the implementation and checks this outcome: hybrid ranking deduplicates entries. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T2-E2E-019 Mem0 identities are isolated for every Cortex workspace |
Creates separate security or workspace scopes and checks this outcome: mem0 identities are isolated for every Cortex workspace. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T2-E2E-020 Mem0 add and search requests carry the workspace-scoped identity |
Creates separate security or workspace scopes and checks this outcome: mem0 add and search requests carry the workspace-scoped identity. | file indexing/broker confinement, search, secret handling, writes, backups, or diffs; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
MBS-1/MBS-8 open_memory_browser uses port 19779 and follows the Matbot lifecycle |
Runs the opt-in Windows lifecycle journey and checks this outcome: open_memory_browser uses port 19779 and follows the Matbot lifecycle. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | L: Default enabled execution and non-Windows hosts; may start/stop services and mutate then restore test secrets. | L: All documented lifecycle opt-ins, a dedicated disposable Windows host, Docker/services, and a fake provider for the provider journey. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T3-E2E-030 standalone memory browser provides health, filtering, CRUD, and version-aware updates |
Runs the focused fixture against the implementation and checks this outcome: standalone memory browser provides health, filtering, CRUD, and version-aware updates. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T3-E2E-039 memory policy declares non-overlapping durable-memory safeguards |
Runs the focused fixture against the implementation and checks this outcome: memory policy declares non-overlapping durable-memory safeguards. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-040 path mappings define complete Windows, WSL, and Docker prefixes for Cortex workspace files |
Runs the focused fixture against the implementation and checks this outcome: path mappings define complete Windows, WSL, and Docker prefixes for Cortex workspace files. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T3-E2E-042 memory recall |
Runs the focused fixture against the implementation and checks this outcome: memory recall. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-042 memory recall › a fact stated in one conversation is recalled in a later, separate conversation |
Runs the focused fixture against the implementation and checks this outcome: a fact stated in one conversation is recalled in a later, separate conversation. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-042 memory recall › recall survives a process restart |
Creates state, crosses the stated persistence or restart boundary, and checks this outcome: recall survives a process restart. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-042 memory recall › one fact is shared by every conversation in the workspace |
Runs the focused fixture against the implementation and checks this outcome: one fact is shared by every conversation in the workspace. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-042 memory recall › facts do not leak from one workspace into another |
Creates separate security or workspace scopes and checks this outcome: facts do not leak from one workspace into another. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-042 memory recall › each workspace recalls only its own facts |
Runs the focused fixture against the implementation and checks this outcome: each workspace recalls only its own facts. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-042 memory recall › recall costs no model call |
Runs the focused fixture against the implementation and checks this outcome: recall costs no model call. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-042 memory recall › capture records which session and message a fact came from |
Runs the focused fixture against the implementation and checks this outcome: capture records which session and message a fact came from. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-042 memory recall › an unrelated question recalls nothing |
Runs the focused fixture against the implementation and checks this outcome: an unrelated question recalls nothing. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-042 memory recall › a shared function word does not drag in an unrelated note |
Runs the focused fixture against the implementation and checks this outcome: a shared function word does not drag in an unrelated note. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-042 memory recall › a distinctive identifier carries its fact when nothing else in the question matches |
Runs the focused fixture against the implementation and checks this outcome: a distinctive identifier carries its fact when nothing else in the question matches. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-042 memory recall › an inflected question recalls a fact stored in another word form |
Runs the focused fixture against the implementation and checks this outcome: an inflected question recalls a fact stored in another word form. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-042 memory recall › a long pasted note stays recallable from a short question |
Runs the focused fixture against the implementation and checks this outcome: a long pasted note stays recallable from a short question. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-042 memory recall › extra context in a query does not lose a match |
Runs the focused fixture against the implementation and checks this outcome: extra context in a query does not lose a match. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-042 memory recall › repeated facts are collapsed to one line of recalled context |
Runs the focused fixture against the implementation and checks this outcome: repeated facts are collapsed to one line of recalled context. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-042 memory recall › a fact with unusable provenance is still recalled |
Runs the focused fixture against the implementation and checks this outcome: a fact with unusable provenance is still recalled. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-042 memory recall › a system-authored turn does not trigger recall |
Runs the focused fixture against the implementation and checks this outcome: a system-authored turn does not trigger recall. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-042 memory recall › contextual_search finds a fact captured in an earlier conversation |
Runs the focused fixture against the implementation and checks this outcome: contextual_search finds a fact captured in an earlier conversation. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T3-E2E-043 production memory runtime is persistent, workspace-isolated, and test-safe |
Creates state, crosses the stated persistence or restart boundary, and checks this outcome: production memory runtime is persistent, workspace-isolated, and test-safe. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
MISSING-05 runtime plugin discovery is stable, deduplicated, and constrained to local roots |
Runs the focused fixture against the implementation and checks this outcome: runtime plugin discovery is stable, deduplicated, and constrained to local roots. | plugin discovery, compatibility, installation, activation, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T3-E2E-034 plugin add persists and activates an approved local plugin, rejects missing paths, and rolls back incompatible plugins |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: plugin add persists and activates an approved local plugin, rejects missing paths, and rolls back incompatible plugins. | plugin discovery, compatibility, installation, activation, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
PDR-3/PDR-7 plugin add installs an npm tarball with pnpm and preserves its dependency graph |
Runs the focused fixture against the implementation and checks this outcome: plugin add installs an npm tarball with pnpm and preserves its dependency graph. | plugin discovery, compatibility, installation, activation, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
PowerShell tool executes a temporary script with cwd, environment, and Bypass policy |
Executes the PowerShell boundary on Windows and checks this outcome: powerShell tool executes a temporary script with cwd, environment, and Bypass policy. | process execution, environment minimization, cwd confinement, timeout, and shell boundaries; specifically the outcome and boundary stated in this title. | Z: Non-Windows execution and non-PowerShell shells; no production scripts or secrets are used. | Z: Windows PowerShell is available and temporary scripts/processes can be created and terminated. |
PowerShell tool keeps cwd and environment values out of shell parsing |
Executes the PowerShell boundary on Windows and checks this outcome: powerShell tool keeps cwd and environment values out of shell parsing. | process execution, environment minimization, cwd confinement, timeout, and shell boundaries; specifically the outcome and boundary stated in this title. | Z: Non-Windows execution and non-PowerShell shells; no production scripts or secrets are used. | Z: Windows PowerShell is available and temporary scripts/processes can be created and terminated. |
PowerShell tool reports a timed-out long-running process as an error and cleans up its script |
Executes the PowerShell boundary on Windows and checks this outcome: powerShell tool reports a timed-out long-running process as an error and cleans up its script. | process execution, environment minimization, cwd confinement, timeout, and shell boundaries; specifically the outcome and boundary stated in this title. | Z: Non-Windows execution and non-PowerShell shells; no production scripts or secrets are used. | Z: Windows PowerShell is available and temporary scripts/processes can be created and terminated. |
PowerShell invocation retains the documented non-interactive, no-profile argument boundary |
Runs the focused fixture against the implementation and checks this outcome: powerShell invocation retains the documented non-interactive, no-profile argument boundary. | process execution, environment minimization, cwd confinement, timeout, and shell boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
question ids are unique and non-empty |
Runs the focused fixture against the implementation and checks this outcome: question ids are unique and non-empty. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
every expectation is a group of alternatives with valid regular expressions |
Runs the focused fixture against the implementation and checks this outcome: every expectation is a group of alternatives with valid regular expressions. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
each supporting quote is still present in README.md |
Runs the focused fixture against the implementation and checks this outcome: each supporting quote is still present in README.md. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
each supporting quote scores 1 against its own question |
Runs the focused fixture against the implementation and checks this outcome: each supporting quote scores 1 against its own question. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
non-answers score 0 |
Runs the focused fixture against the implementation and checks this outcome: non-answers score 0. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
summary arithmetic is binary and exact |
Runs the focused fixture against the implementation and checks this outcome: summary arithmetic is binary and exact. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
runtime reliability boundaries reject, recover, and shut down without crashing or hanging |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: runtime reliability boundaries reject, recover, and shut down without crashing or hanging. | runtime reliability behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T3-E2E-005 / MISSING-10 scheduled execution records one policy-bound occurrence with inherited identity |
Runs the focused fixture against the implementation and checks this outcome: scheduled execution records one policy-bound occurrence with inherited identity. | governed workflows, approvals, scheduling, or structured-data execution; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
hardened services bind loopback, enforce Host checks, tokens, and 400 mapping |
Runs the focused fixture against the implementation and checks this outcome: hardened services bind loopback, enforce Host checks, tokens, and 400 mapping. | loopback HTTP service security, protocol, limits, shutdown, and error handling; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
file-broker enforces the shared-secret token when configured |
Runs the focused fixture against the implementation and checks this outcome: file-broker enforces the shared-secret token when configured. | loopback HTTP service security, protocol, limits, shutdown, and error handling; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
file-index binds loopback, rejects foreign hosts, and enforces its token |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: file-index binds loopback, rejects foreign hosts, and enforces its token. | loopback HTTP service security, protocol, limits, shutdown, and error handling; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
broker writes are atomic: no temp residue and never partial content |
Runs the focused fixture against the implementation and checks this outcome: broker writes are atomic: no temp residue and never partial content. | loopback HTTP service security, protocol, limits, shutdown, and error handling; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
sendJsonError logs internal errors server-side and returns a generic 500 message |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: sendJsonError logs internal errors server-side and returns a generic 500 message. | loopback HTTP service security, protocol, limits, shutdown, and error handling; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
requestAbortSignal removes its listeners when the request closes normally or aborts |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: requestAbortSignal removes its listeners when the request closes normally or aborts. | loopback HTTP service security, protocol, limits, shutdown, and error handling; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
MISSING-13 source freshness transitions at the SLA boundary without real-time waits |
Runs the focused fixture against the implementation and checks this outcome: source freshness transitions at the SLA boundary without real-time waits. | architecture sources, connectors, graph evidence, provenance, or health; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T3-E2E-033 source-registry runtime flow covers versions, access history, and health transitions |
Runs the focused fixture against the implementation and checks this outcome: source-registry runtime flow covers versions, access history, and health transitions. | architecture sources, connectors, graph evidence, provenance, or health; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
sqlite put rejects streams exceeding maxBytes while streaming |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: sqlite put rejects streams exceeding maxBytes while streaming. | filesystem/SQLite storage, CAS, concurrency, migration, watch, and size boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
sqlite distinct-content equal-size writes yield distinct CAS versions |
Runs the focused fixture against the implementation and checks this outcome: sqlite distinct-content equal-size writes yield distinct CAS versions. | filesystem/SQLite storage, CAS, concurrency, migration, watch, and size boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
sqlite watch returns promptly on a pre-aborted signal |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: sqlite watch returns promptly on a pre-aborted signal. | filesystem/SQLite storage, CAS, concurrency, migration, watch, and size boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
filesystem set serialises concurrent writes through the per-id lock |
Runs the focused fixture against the implementation and checks this outcome: filesystem set serialises concurrent writes through the per-id lock. | filesystem/SQLite storage, CAS, concurrency, migration, watch, and size boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
skills-node watcher retries transient readdir failures |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: skills-node watcher retries transient readdir failures. | filesystem/SQLite storage, CAS, concurrency, migration, watch, and size boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
skills-node watcher creates a missing skills dir at startup |
Runs the focused fixture against the implementation and checks this outcome: skills-node watcher creates a missing skills dir at startup. | filesystem/SQLite storage, CAS, concurrency, migration, watch, and size boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
REL-03 failed stream preserves contents and metadata across restart |
Uses temporary file stores with interrupted writes, concurrent mutations, legacy entries, and aged blobs. | Atomic payload/metadata publication, restart consistency, stable handles, legacy reads, and safe orphan retention. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-03 failed blob-rename preserves contents and metadata across restart |
Uses temporary file stores with interrupted writes, concurrent mutations, legacy entries, and aged blobs. | Atomic payload/metadata publication, restart consistency, stable handles, legacy reads, and safe orphan retention. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-03 failed manifest-write preserves contents and metadata across restart |
Uses temporary file stores with interrupted writes, concurrent mutations, legacy entries, and aged blobs. | Atomic payload/metadata publication, restart consistency, stable handles, legacy reads, and safe orphan retention. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-03 failed manifest-rename preserves contents and metadata across restart |
Uses temporary file stores with interrupted writes, concurrent mutations, legacy entries, and aged blobs. | Atomic payload/metadata publication, restart consistency, stable handles, legacy reads, and safe orphan retention. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-03 manifests publish consistent versions, preserve names, and serialize put/delete across instances |
Uses temporary file stores with interrupted writes, concurrent mutations, legacy entries, and aged blobs. | Atomic payload/metadata publication, restart consistency, stable handles, legacy reads, and safe orphan retention. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-03 cleanup retains referenced versions and corrupt metadata is reported |
Uses temporary file stores with interrupted writes, concurrent mutations, legacy entries, and aged blobs. | Atomic payload/metadata publication, restart consistency, stable handles, legacy reads, and safe orphan retention. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-03 legacy named and anonymous entries remain readable and migrate on overwrite |
Uses temporary file stores with interrupted writes, concurrent mutations, legacy entries, and aged blobs. | Atomic payload/metadata publication, restart consistency, stable handles, legacy reads, and safe orphan retention. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
E2E-015 / T3-E2E-015 structured-data runtime enforces approval expiry and governed SQL boundaries |
Runs the focused fixture against the implementation and checks this outcome: structured-data runtime enforces approval expiry and governed SQL boundaries. | governed workflows, approvals, scheduling, or structured-data execution; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
MISSING-00 deterministic test helpers isolate paths, time, and sidecar traffic |
Creates separate security or workspace scopes and checks this outcome: deterministic test helpers isolate paths, time, and sidecar traffic. | test helpers behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
successful tool results serialize unchanged (no is_error marker) |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: successful tool results serialize unchanged (no is_error marker). | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
error tool results carry an explicit is_error flag on the wire (spec R4) |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: error tool results carry an explicit is_error flag on the wire (spec R4). | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
non-object error results are wrapped so is_error always survives |
Creates state, crosses the stated persistence or restart boundary, and checks this outcome: non-object error results are wrapped so is_error always survives. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
openai-compat adapter surfaces unparseable tool arguments as failed calls instead of throwing |
Runs the focused fixture against the implementation and checks this outcome: openai-compat adapter surfaces unparseable tool arguments as failed calls instead of throwing. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
globToRegExp handles **, *, ?, and classes |
Runs the focused fixture against the implementation and checks this outcome: globToRegExp handles **, *, ?, and classes. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
summarizeDiff counts additions and deletions with context |
Runs the focused fixture against the implementation and checks this outcome: summarizeDiff counts additions and deletions with context. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
read returns numbered lines with offset/limit and continuation hints |
Runs the focused fixture against the implementation and checks this outcome: read returns numbered lines with offset/limit and continuation hints. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
read enforces byte caps and refuses missing/binary/directory paths helpfully |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: read enforces byte caps and refuses missing/binary/directory paths helpfully. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
read rejects relative paths and escapes outside the workspace root |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: read rejects relative paths and escapes outside the workspace root. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
write creates new files (with parent directories) and reports creation |
Runs the focused fixture against the implementation and checks this outcome: write creates new files (with parent directories) and reports creation. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
write refuses files changed on disk since the last read (stale-write guard) |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: write refuses files changed on disk since the last read (stale-write guard). | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
edit replaces unique text, reports a diff, and preserves CRLF endings |
Runs the focused fixture against the implementation and checks this outcome: edit replaces unique text, reports a diff, and preserves CRLF endings. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
edit fails on multiple matches without replaceAll and on unknown text |
Runs the focused fixture against the implementation and checks this outcome: edit fails on multiple matches without replaceAll and on unknown text. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
edit redirects empty oldString to write and requires prior reads |
Runs the focused fixture against the implementation and checks this outcome: edit redirects empty oldString to write and requires prior reads. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
glob matches patterns, prunes ignored dirs, and caps results |
Runs the focused fixture against the implementation and checks this outcome: glob matches patterns, prunes ignored dirs, and caps results. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
grep finds regex matches with line numbers and validates patterns |
Runs the focused fixture against the implementation and checks this outcome: grep finds regex matches with line numbers and validates patterns. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
list renders a pruned depth-limited tree |
Runs the focused fixture against the implementation and checks this outcome: list renders a pruned depth-limited tree. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
todowrite stores validated lists per session and emits durable markers |
Runs the focused fixture against the implementation and checks this outcome: todowrite stores validated lists per session and emits durable markers. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
schema validator accepts conforming input and reports precise issues |
Runs the focused fixture against the implementation and checks this outcome: schema validator accepts conforming input and reports precise issues. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
schema validator supports anyOf, $ref, additionalProperties:false, patterns |
Runs the focused fixture against the implementation and checks this outcome: schema validator supports anyOf, $ref, additionalProperties:false, patterns. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
formatValidationIssues produces corrective model-facing text |
Runs the focused fixture against the implementation and checks this outcome: formatValidationIssues produces corrective model-facing text. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
wildcard matching handles *, **, and ? segments |
Runs the focused fixture against the implementation and checks this outcome: wildcard matching handles *, **, and ? segments. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
evaluatePermission uses last matching rule with fallback |
Runs the focused fixture against the implementation and checks this outcome: evaluatePermission uses last matching rule with fallback. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
truncateToolResult caps oversized strings with head/tail preview |
Runs the focused fixture against the implementation and checks this outcome: truncateToolResult caps oversized strings with head/tail preview. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
truncateToolResult saves full output through the FileStore when available |
Runs the focused fixture against the implementation and checks this outcome: truncateToolResult saves full output through the FileStore when available. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
invalid tool inputs are rejected with corrective errors before execution (R3) |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: invalid tool inputs are rejected with corrective errors before execution (R3). | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
parseError tool calls degrade to failed calls instead of aborting the turn (R4) |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: parseError tool calls degrade to failed calls instead of aborting the turn (R4). | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
loop policy stops runaway turns gracefully at maxIterations (R1) |
Runs the focused fixture against the implementation and checks this outcome: loop policy stops runaway turns gracefully at maxIterations (R1). | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
loop policy answers excess tool calls with budget errors at maxToolCallsPerTurn (R1) |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: loop policy answers excess tool calls with budget errors at maxToolCallsPerTurn (R1). | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
parallel tool calls execute concurrently unless marked serial (R2) |
Runs the focused fixture against the implementation and checks this outcome: parallel tool calls execute concurrently unless marked serial (R2). | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
doom-loop interception blocks repeated identical calls (R6) |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: doom-loop interception blocks repeated identical calls (R6). | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
hook-aborted turns persist paired tool-call results (R19) |
Creates state, crosses the stated persistence or restart boundary, and checks this outcome: hook-aborted turns persist paired tool-call results (R19). | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
configured deny rules produce permission_denied results without execution |
Runs the focused fixture against the implementation and checks this outcome: configured deny rules produce permission_denied results without execution. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
ask rules prompt via PromptFn; allow proceeds, always approves future calls |
Runs the focused fixture against the implementation and checks this outcome: ask rules prompt via PromptFn; allow proceeds, always approves future calls. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
denied permission asks feed rejection feedback back to the model |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: denied permission asks feed rejection feedback back to the model. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
fully denied tools are absent from the advertised tool menu |
Runs the focused fixture against the implementation and checks this outcome: fully denied tools are absent from the advertised tool menu. | tool runner, harness tools, permission gates, or provider-wire adaptation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
E2E-023 user guide relative links and heading anchors resolve |
Reads the shipped source, configuration, or documentation and checks this contract: user guide relative links and heading anchors resolve. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | S: Runtime execution of the documented feature and behavior outside the inspected files or literal contract. | S: The checked-in documents, configuration, paths, labels, and source literals are the authoritative shipped contract. |
user guide commands, port, and important WebUI labels match shipped files |
Reads the shipped source, configuration, or documentation and checks this contract: user guide commands, port, and important WebUI labels match shipped files. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | S: Runtime execution of the documented feature and behavior outside the inspected files or literal contract. | S: The checked-in documents, configuration, paths, labels, and source literals are the authoritative shipped contract. |
user guide retains explicit product limitations and destructive warnings |
Reads the shipped source, configuration, or documentation and checks this contract: user guide retains explicit product limitations and destructive warnings. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | S: Runtime execution of the documented feature and behavior outside the inspected files or literal contract. | S: The checked-in documents, configuration, paths, labels, and source literals are the authoritative shipped contract. |
user guide localhost safety boundary matches the production listener |
Reads the shipped source, configuration, or documentation and checks this contract: user guide localhost safety boundary matches the production listener. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | S: Runtime execution of the documented feature and behavior outside the inspected files or literal contract. | S: The checked-in documents, configuration, paths, labels, and source literals are the authoritative shipped contract. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
MISSING-15 branding configuration validates safe defaults and presentation tokens |
Runs the focused fixture against the implementation and checks this outcome: branding configuration validates safe defaults and presentation tokens. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
WebUI honours MATBOT_WEB_PORT and serves install-scoped branding on that listener |
Creates separate security or workspace scopes and checks this outcome: webUI honours MATBOT_WEB_PORT and serves install-scoped branding on that listener. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
execution-class tools are denied over HTTP unless explicitly opted in |
Runs the focused fixture against the implementation and checks this outcome: execution-class tools are denied over HTTP unless explicitly opted in. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
shell-tool guard lifts with CORTEX_WEBUI_ALLOW_SHELL_TOOLS=1 |
Runs the focused fixture against the implementation and checks this outcome: shell-tool guard lifts with CORTEX_WEBUI_ALLOW_SHELL_TOOLS=1. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
CORS reflects only loopback origins; foreign origins get no ACAO header |
Runs the focused fixture against the implementation and checks this outcome: cORS reflects only loopback origins; foreign origins get no ACAO header. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
foreign or missing Host headers are rejected with 403 (DNS rebinding) |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: foreign or missing Host headers are rejected with 403 (DNS rebinding). | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
CORTEX_WEBUI_TOKEN enforces x-cortex-token on mutating routes only |
Runs the focused fixture against the implementation and checks this outcome: cORTEX_WEBUI_TOKEN enforces x-cortex-token on mutating routes only. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
contended session appends retry CAS and return 409 instead of bypassing it |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: contended session appends retry CAS and return 409 instead of bypassing it. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
session appends succeed once CAS contention clears; no message loss |
Runs the focused fixture against the implementation and checks this outcome: session appends succeed once CAS contention clears; no message loss. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
oversized request bodies are destroyed instead of buffered |
Runs the focused fixture against the implementation and checks this outcome: oversized request bodies are destroyed instead of buffered. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
/index.html serves the static UI like / does (route typo fixed) |
Runs the focused fixture against the implementation and checks this outcome: /index.html serves the static UI like / does (route typo fixed). | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
WCD-1/2/3/5/6/7/8 draft-only compilation persists deterministically and can later publish |
Creates state, crosses the stated persistence or restart boundary, and checks this outcome: 2/3/5/6/7/8 draft-only compilation persists deterministically and can later publish. | governed workflows, approvals, scheduling, or structured-data execution; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
WCD-4 draft compilation persists validation errors and compiler warnings for inspection |
Creates state, crosses the stated persistence or restart boundary, and checks this outcome: draft compilation persists validation errors and compiler warnings for inspection. | governed workflows, approvals, scheduling, or structured-data execution; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
E2E-005 / MISSING-08 workflow-governance runtime flow covers shadow labels and approval ledgers |
Runs the focused fixture against the implementation and checks this outcome: workflow-governance runtime flow covers shadow labels and approval ledgers. | governed workflows, approvals, scheduling, or structured-data execution; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T3-E2E-003 workspace deletion is ownership-safe and protects the active workspace |
Runs the focused fixture against the implementation and checks this outcome: workspace deletion is ownership-safe and protects the active workspace. | workspace/process lifecycle, switch fencing, cleanup, and isolation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
MISSING-02 adaptive file concurrency runs bulk backlogs wide and incremental scans sequentially |
Runs the focused fixture against the implementation and checks this outcome: adaptive file concurrency runs bulk backlogs wide and incremental scans sequentially. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
MISSING-02 adaptive file concurrency keeps small scans strictly sequential |
Runs the focused fixture against the implementation and checks this outcome: adaptive file concurrency keeps small scans strictly sequential. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
MISSING-03/MISSING-12 RAG migrates legacy config and rejects blank, duplicate, or inaccessible context settings |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: rAG migrates legacy config and rejects blank, duplicate, or inaccessible context settings. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
CUDA sidecar handles CPU detection, model reuse, E5 preprocessing, and GPU OOM without a GPU |
Launches or inspects the local Python sidecar fixture and checks this outcome: cUDA sidecar handles CPU detection, model reuse, E5 preprocessing, and GPU OOM without a GPU. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | Y: A real GPU, downloaded production model guarantees, live providers, and remote sidecars. | Y: A usable Python interpreter and the sidecar's local dependencies; the test is skipped when Python is absent. |
CUDA image preloads models and mounts a persistent Hugging Face cache |
Launches or inspects the local Python sidecar fixture and checks this outcome: cUDA image preloads models and mounts a persistent Hugging Face cache. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | Y: A real GPU, downloaded production model guarantees, live providers, and remote sidecars. | Y: A usable Python interpreter and the sidecar's local dependencies; the test is skipped when Python is absent. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
CUDA launch failures retain the sidecar diagnostic and safely use CPU embeddings |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: cUDA launch failures retain the sidecar diagnostic and safely use CPU embeddings. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
MISSING-01 CUDA embedding health validates model/profile/dimension/prefix contracts |
Runs the focused fixture against the implementation and checks this outcome: cUDA embedding health validates model/profile/dimension/prefix contracts. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
MISSING-01 runtime sidecar uses E5 prefixes, dimensions, and document/query batches |
Runs the focused fixture against the implementation and checks this outcome: runtime sidecar uses E5 prefixes, dimensions, and document/query batches. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T3-E2E-031 workspace-rag sends purpose-aware requests and persists the embedding signature |
Creates state, crosses the stated persistence or restart boundary, and checks this outcome: workspace-rag sends purpose-aware requests and persists the embedding signature. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
MISSING-02 embedding responses are validated against the launch contract, not the request |
Runs the focused fixture against the implementation and checks this outcome: embedding responses are validated against the launch contract, not the request. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
regex validator rejects nested quantifier groups while keeping safe patterns |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: regex validator rejects nested quantifier groups while keeping safe patterns. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
memory repository regex evaluation terminates a catastrophic pattern within its time budget |
Runs the focused fixture against the implementation and checks this outcome: memory repository regex evaluation terminates a catastrophic pattern within its time budget. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
rate limiter releases reserved budget when a wait is aborted and rejects pre-aborted signals |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: rate limiter releases reserved budget when a wait is aborted and rejects pre-aborted signals. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
line index writer surfaces stream errors instead of crashing the process |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: line index writer surfaces stream errors instead of crashing the process. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
manager close() stops the lazy embedding worker before closing the repository |
Runs the focused fixture against the implementation and checks this outcome: manager close() stops the lazy embedding worker before closing the repository. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
evidence assembly fetches ranges by hit.contentSha256 end to end |
Runs the focused fixture against the implementation and checks this outcome: evidence assembly fetches ranges by hit.contentSha256 end to end. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
MISSING-02 ingestion throughput policy resolves defaults, overrides, and clamps |
Runs the focused fixture against the implementation and checks this outcome: ingestion throughput policy resolves defaults, overrides, and clamps. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
reranker service enforces bounds, reuses one model, and degrades GPU OOM |
Launches or inspects the local Python sidecar fixture and checks this outcome: reranker service enforces bounds, reuses one model, and degrades GPU OOM. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | Y: A real GPU, downloaded production model guarantees, live providers, and remote sidecars. | Y: A usable Python interpreter and the sidecar's local dependencies; the test is skipped when Python is absent. |
reranker Compose service is optional and shares the persistent model cache |
Creates state, crosses the stated persistence or restart boundary, and checks this outcome: reranker Compose service is optional and shares the persistent model cache. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
MISSING-01/MISSING-03 V2 RAG reports reconciliation, queue, and safe failure states |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: rAG reports reconciliation, queue, and safe failure states. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
workspace RAG V2 evaluation calculates graded retrieval, citation, and ACL metrics |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 evaluation calculates graded retrieval, citation, and ACL metrics. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 evaluation detects invalid citations and forbidden evidence |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: workspace RAG V2 evaluation detects invalid citations and forbidden evidence. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
orphan GC environment settings reject invalid values with documented defaults |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: orphan GC environment settings reject invalid values with documented defaults. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
orphan GC reclaims removed-folder documents and records cleanup status |
Runs the focused fixture against the implementation and checks this outcome: orphan GC reclaims removed-folder documents and records cleanup status. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
managed blob GC retains referenced and fresh objects while deleting capped stale orphans |
Runs the focused fixture against the implementation and checks this outcome: managed blob GC retains referenced and fresh objects while deleting capped stale orphans. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
orphan GC honors busy, staging-publication, and grace-period safety gates |
Runs the focused fixture against the implementation and checks this outcome: orphan GC honors busy, staging-publication, and grace-period safety gates. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
context purge is idempotent and retains retrieval audit evidence |
Runs the focused fixture against the implementation and checks this outcome: context purge is idempotent and retains retrieval audit evidence. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
workspace RAG V2 publishes a three-level hybrid index and returns verified evidence |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 publishes a three-level hybrid index and returns verified evidence. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 census is bounded, resumable, and forecasts tiered storage without embeddings |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 census is bounded, resumable, and forecasts tiered storage without embeddings. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 promotes routing metadata and enforces document, jurisdiction, and as-of filters |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 promotes routing metadata and enforces document, jurisdiction, and as-of filters. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 backfill streams authoritative and current sources before archives |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 backfill streams authoritative and current sources before archives. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 cancellation preserves the active publication |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: workspace RAG V2 cancellation preserves the active publication. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 resumes the generation an interrupted run left behind |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 resumes the generation an interrupted run left behind. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 checkpoint publications leave an interrupted scan queryable |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 checkpoint publications leave an interrupted scan queryable. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 prunes staging generations abandoned by earlier runs |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 prunes staging generations abandoned by earlier runs. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 keeps complete lexical coverage and lazily promotes cold passages |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 keeps complete lexical coverage and lazily promotes cold passages. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 caps planned passage vectors for medium files |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 caps planned passage vectors for medium files. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 reuses content-addressed derivatives across duplicate sources |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 reuses content-addressed derivatives across duplicate sources. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 reconciles forced edits, path-based renames, deletions, and source lifecycle |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 reconciles forced edits, path-based renames, deletions, and source lifecycle. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 does not publish deletions when a configured root is unavailable |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: workspace RAG V2 does not publish deletions when a configured root is unavailable. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 counts the corpus before scanning and reports documents held in the database |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 counts the corpus before scanning and reports documents held in the database. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 skips an invalid configured root and indexes the remaining roots |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: workspace RAG V2 skips an invalid configured root and indexes the remaining roots. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 does not publish nested-directory deletions from a discovery race |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 does not publish nested-directory deletions from a discovery race. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 defers deletions when any discovered file fails ingestion |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 defers deletions when any discovered file fails ingestion. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 derivative reuse hashes the exact path-derived embedding input |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 derivative reuse hashes the exact path-derived embedding input. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 builds an unchanged corpus as a side-by-side derivative generation when the embedding signature changes |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 builds an unchanged corpus as a side-by-side derivative generation when the embedding signature changes. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 regenerates unchanged semantic derivatives when the summarizer signature changes |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 regenerates unchanged semantic derivatives when the summarizer signature changes. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 reciprocal rank fusion preserves retriever reasons |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 reciprocal rank fusion preserves retriever reasons. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 rewrites conversational follow-ups and decomposes comparison and diagnostic searches |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 rewrites conversational follow-ups and decomposes comparison and diagnostic searches. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 performs iterative retrieval and explicitly abstains without evidence |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 performs iterative retrieval and explicitly abstains without evidence. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 creates versioned semantic routing summaries and routes a split book through its collection |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 creates versioned semantic routing summaries and routes a split book through its collection. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
REL-02 failed rebuildCollections cannot mutate an active checkpoint |
Fails final staging collection rebuild, validation, or publication after a checkpoint. | Independent active snapshots and searchable evidence while staging applies deletions. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-02 failed validateGeneration cannot mutate an active checkpoint |
Fails final staging collection rebuild, validation, or publication after a checkpoint. | Independent active snapshots and searchable evidence while staging applies deletions. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-02 failed publishGeneration cannot mutate an active checkpoint |
Fails final staging collection rebuild, validation, or publication after a checkpoint. | Independent active snapshots and searchable evidence while staging applies deletions. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-05 full summary queues cancel and close without waiting for the summarizer |
Fills the summary queue and holds summarizers past cancellation or their deadline. | Abortable producers, bounded shutdown, queue draining, and prevention of late writes. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-05 summary deadline drains even an uncooperative summarizer |
Fills the summary queue and holds summarizers past cancellation or their deadline. | Abortable producers, bounded shutdown, queue draining, and prevention of late writes. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-10 initialize setup failure is terminal and observable |
Fails ingestion setup or terminal persistence, cancels setup, then restores the repository. | Terminal job status, visible outage details, no active work after failure, and recovery of orphaned jobs. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-10 createJob setup failure is terminal and observable |
Fails ingestion setup or terminal persistence, cancels setup, then restores the repository. | Terminal job status, visible outage details, no active work after failure, and recovery of orphaned jobs. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-10 listFingerprints setup failure is terminal and observable |
Fails ingestion setup or terminal persistence, cancels setup, then restores the repository. | Terminal job status, visible outage details, no active work after failure, and recovery of orphaned jobs. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-10 availableMarkdownRoots setup failure is terminal and observable |
Fails ingestion setup or terminal persistence, cancels setup, then restores the repository. | Terminal job status, visible outage details, no active work after failure, and recovery of orphaned jobs. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-10 countMarkdown setup failure is terminal and observable |
Fails ingestion setup or terminal persistence, cancels setup, then restores the repository. | Terminal job status, visible outage details, no active work after failure, and recovery of orphaned jobs. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-10 beginGeneration setup failure is terminal and observable |
Fails ingestion setup or terminal persistence, cancels setup, then restores the repository. | Terminal job status, visible outage details, no active work after failure, and recovery of orphaned jobs. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-10 pruneStagingGenerations setup failure is terminal and observable |
Fails ingestion setup or terminal persistence, cancels setup, then restores the repository. | Terminal job status, visible outage details, no active work after failure, and recovery of orphaned jobs. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-10 cancellation during setup persists a cancelled terminal job |
Fails ingestion setup or terminal persistence, cancels setup, then restores the repository. | Terminal job status, visible outage details, no active work after failure, and recovery of orphaned jobs. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
REL-10 repository outage retains failure details and later ingestion recovers orphaned work |
Fails ingestion setup or terminal persistence, cancels setup, then restores the repository. | Terminal job status, visible outage details, no active work after failure, and recovery of orphaned jobs. | Live model services, production stores, process-wide crash/power-loss durability, and multi-host scheduling. | Node 24+, temporary files and loopback access; deterministic injected failures and faithful test doubles. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
workspace RAG V2 OpenSearch adapter remains an explicit derivative backend |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 OpenSearch adapter remains an explicit derivative backend. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
OpenSearch promotion requires a measured PostgreSQL trigger and every safety gate |
Runs the focused fixture against the implementation and checks this outcome: openSearch promotion requires a measured PostgreSQL trigger and every safety gate. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
measurement-gated ColBERT adapter preserves scope and returns manifest-bearing candidates |
Creates separate security or workspace scopes and checks this outcome: measurement-gated ColBERT adapter preserves scope and returns manifest-bearing candidates. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
workspace RAG V2 is the only index mode and defaults to primary |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 is the only index mode and defaults to primary. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 streams hierarchy with immutable byte and line citations |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 streams hierarchy with immutable byte and line citations. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 splits a very long structural unit within the parser memory budget |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 splits a very long structural unit within the parser memory budget. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 promotes front matter, preserves fenced blocks, and repeats split table headers only for search |
Feeds the stated invalid or unsafe boundary through the implementation and checks this outcome: workspace RAG V2 promotes front matter, preserves fenced blocks, and repeats split table headers only for search. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 object retention supports managed, external immutable, and explicit manifest-only modes |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 object retention supports managed, external immutable, and explicit manifest-only modes. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 parses a sparse 2 GiB source with bounded buffers |
Creates a sparse 2-GiB Markdown source, streams it, and asserts bounded parsing and the tail/title result. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | G: Default enabled execution and dense real-world 2-GiB content; the fixture is sparse. | G: CORTEX_RAG_V2_2GB_INTEGRATION=1, sparse-file support, enough address space, disk quota, and time. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
workspace_rag is V2-only, auto-reconciles configured folders, and exposes unified status |
Runs the focused fixture against the implementation and checks this outcome: workspace_rag is V2-only, auto-reconciles configured folders, and exposes unified status. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace_rag off mode disables V2 commands without exposing an alternate index |
Runs the focused fixture against the implementation and checks this outcome: workspace_rag off mode disables V2 commands without exposing an alternate index. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
workspace RAG V2 PostgreSQL publishes lexical and pgvector generations |
Runs the V2 repository against real PostgreSQL/pgvector and checks this outcome: workspace RAG V2 PostgreSQL publishes lexical and pgvector generations. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | P: Default enabled execution, non-PostgreSQL backends, live providers, and browser behavior. | P: CORTEX_RAG_V2_POSTGRES_INTEGRATION=1, reachable pgvector PostgreSQL, disposable schemas, and sufficient role rights for RLS coverage. |
workspace RAG V2 PostgreSQL resumes an interrupted scan and prunes abandoned generations |
Runs the V2 repository against real PostgreSQL/pgvector and checks this outcome: workspace RAG V2 PostgreSQL resumes an interrupted scan and prunes abandoned generations. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | P: Default enabled execution, non-PostgreSQL backends, live providers, and browser behavior. | P: CORTEX_RAG_V2_POSTGRES_INTEGRATION=1, reachable pgvector PostgreSQL, disposable schemas, and sufficient role rights for RLS coverage. |
workspace RAG V2 enforces RLS through a separate non-owner application role |
Runs the V2 repository against real PostgreSQL/pgvector and checks this outcome: workspace RAG V2 enforces RLS through a separate non-owner application role. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | P: Default enabled execution, non-PostgreSQL backends, live providers, and browser behavior. | P: CORTEX_RAG_V2_POSTGRES_INTEGRATION=1, reachable pgvector PostgreSQL, disposable schemas, and sufficient role rights for RLS coverage. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
workspace RAG V2 multilingual reranker reorders a bounded fused candidate set |
Runs the focused fixture against the implementation and checks this outcome: workspace RAG V2 multilingual reranker reorders a bounded fused candidate set. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
workspace RAG V2 keeps fused results when the reranker is unavailable |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: workspace RAG V2 keeps fused results when the reranker is unavailable. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T2-E2E-009 / T3-E2E-011 / MISSING-03 / MISSING-12 workspace-rag normalizes paths, reconciles reindexing, and manages isolated contexts |
Creates separate security or workspace scopes and checks this outcome: workspace-rag normalizes paths, reconciles reindexing, and manages isolated contexts. | Workspace RAG parsing, ingestion, retrieval, storage, policy, or lifecycle; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
T3-E2E-032 workspace switch handoff |
Runs the focused fixture against the implementation and checks this outcome: workspace switch handoff. | workspace/process lifecycle, switch fencing, cleanup, and isolation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-032 workspace switch handoff › the workspace listing identifies the process that answered it |
Runs the focused fixture against the implementation and checks this outcome: the workspace listing identifies the process that answered it. | workspace/process lifecycle, switch fencing, cleanup, and isolation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-032 workspace switch handoff › two runtimes report different identities |
Runs the focused fixture against the implementation and checks this outcome: two runtimes report different identities. | workspace/process lifecycle, switch fencing, cleanup, and isolation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-032 workspace switch handoff › shutdown releases the port while a request is stuck mid-body |
Runs the focused fixture against the implementation and checks this outcome: shutdown releases the port while a request is stuck mid-body. | workspace/process lifecycle, switch fencing, cleanup, and isolation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
T3-E2E-032 workspace switch handoff › shutdown releases the port while an event stream is open |
Runs the focused fixture against the implementation and checks this outcome: shutdown releases the port while an event stream is open. | workspace/process lifecycle, switch fencing, cleanup, and isolation; specifically the outcome and boundary stated in this title. | N: Live model providers, a real browser, Docker/GPU, and production workspace data. | N: Node >=24, repo-root execution, temp-filesystem and loopback access, and faithful declared fakes/stubs. |
Runner: npm run test:cli. Titles in this section: 5.
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
frontend attachment context follows screen retrieval context and is not persisted |
Creates state, crosses the stated persistence or restart boundary, and checks this outcome: frontend attachment context follows screen retrieval context and is not persisted. | attachment ephemeral behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | C: WebUI behavior, live providers, and external services; stores and modules are isolated fixtures. | C: Node >=24, pnpm, the Matbot TypeScript loader, and writable temporary storage. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
startup mode (skip) logs and skips a non-plugin module rather than aborting |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: startup mode (skip) logs and skips a non-plugin module rather than aborting. | loader behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | C: WebUI behavior, live providers, and external services; stores and modules are isolated fixtures. | C: Node >=24, pnpm, the Matbot TypeScript loader, and writable temporary storage. |
interactive mode (throw) surfaces a typed NotAPluginError |
Injects the stated failure, cancellation, or retry boundary and checks this outcome: interactive mode (throw) surfaces a typed NotAPluginError. | loader behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | C: WebUI behavior, live providers, and external services; stores and modules are isolated fixtures. | C: Node >=24, pnpm, the Matbot TypeScript loader, and writable temporary storage. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
ephemeral stores never open or write to a configured persistent backend |
Creates state, crosses the stated persistence or restart boundary, and checks this outcome: ephemeral stores never open or write to a configured persistent backend. | filesystem/SQLite storage, CAS, concurrency, migration, watch, and size boundaries; specifically the outcome and boundary stated in this title. | C: WebUI behavior, live providers, and external services; stores and modules are isolated fixtures. | C: Node >=24, pnpm, the Matbot TypeScript loader, and writable temporary storage. |
the same memory id resolves independently in each workspace data directory |
Runs the focused fixture against the implementation and checks this outcome: the same memory id resolves independently in each workspace data directory. | filesystem/SQLite storage, CAS, concurrency, migration, watch, and size boundaries; specifically the outcome and boundary stated in this title. | C: WebUI behavior, live providers, and external services; stores and modules are isolated fixtures. | C: Node >=24, pnpm, the Matbot TypeScript loader, and writable temporary storage. |
Runner: npm run test:webui. Titles in this section: 131.
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
STE-1 through STE-8 skill trigger modal configures the shipped classifier behaviors and persists them |
Drives the production static WebUI in Chromium and checks this outcome: through STE-8 skill trigger modal configures the shipped classifier behaviors and persists them. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
plugin details show description, runtime types, tools, and incompatible runtime requirements |
Drives the production static WebUI in Chromium and checks this outcome: plugin details show description, runtime types, tools, and incompatible runtime requirements. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
AT-1 through AT-5 architecture tabs provide roving focus, native activation, and persistent URL routing |
Drives the production static WebUI in Chromium and checks this outcome: through AT-5 architecture tabs provide roving focus, native activation, and persistent URL routing. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
SHP implemented source-health detail keeps unhealthy findings and events attached to the selected source |
Drives the production static WebUI in Chromium and checks this outcome: sHP implemented source-health detail keeps unhealthy findings and events attached to the selected source. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
SHP-3 critical source findings use the critical list and count treatment |
Drives the production static WebUI in Chromium and checks this outcome: critical source findings use the critical list and count treatment. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
GSP governed SQL keeps execution disabled until approval and renders the planned query record |
Drives the production static WebUI in Chromium and checks this outcome: gSP governed SQL keeps execution disabled until approval and renders the planned query record. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
WOC overview and run ledger expose summary metrics, typed inputs, evidence, events, and filters |
Drives the production static WebUI in Chromium and checks this outcome: wOC overview and run ledger expose summary metrics, typed inputs, evidence, events, and filters. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
WOC-8 workflow overview refreshes automatically every 30 seconds while visible |
Drives the production static WebUI in Chromium and checks this outcome: workflow overview refreshes automatically every 30 seconds while visible. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
WOC-5 failed recent runs use the critical visual treatment |
Drives the production static WebUI in Chromium and checks this outcome: failed recent runs use the critical visual treatment. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
HWA-2/HWA-3 high-risk workflow write opens an approval modal with warnings and a unified diff |
Drives the production static WebUI in Chromium and checks this outcome: high-risk workflow write opens an approval modal with warnings and a unified diff. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
WCD-4 draft library displays compiler validation errors and warnings |
Drives the production static WebUI in Chromium and checks this outcome: draft library displays compiler validation errors and warnings. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
WSL shadow outcome controls record accept, reject, and mixed labels and recalculate readiness |
Drives the production static WebUI in Chromium and checks this outcome: wSL shadow outcome controls record accept, reject, and mixed labels and recalculate readiness. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
ERO evaluation panel renders the waterfall contract and sponsor evidence aggregates |
Drives the production static WebUI in Chromium and checks this outcome: eRO evaluation panel renders the waterfall contract and sponsor evidence aggregates. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
Workspace RAG V2 generated-corpus acceptance › publishes complete lexical coverage and a bounded hierarchical index |
Drives the production static WebUI in Chromium and checks this outcome: publishes complete lexical coverage and a bounded hierarchical index. | hybrid retrieval behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
Workspace RAG V2 generated-corpus acceptance › streams a varied corpus and reports structures, duplicates, tiers, and forecasts |
Drives the production static WebUI in Chromium and checks this outcome: streams a varied corpus and reports structures, duplicates, tiers, and forecasts. | hybrid retrieval behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
Workspace RAG V2 generated-corpus acceptance › routes document to section to passage and fuses exact, lexical, and dense lanes |
Drives the production static WebUI in Chromium and checks this outcome: routes document to section to passage and fuses exact, lexical, and dense lanes. | hybrid retrieval behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
Workspace RAG V2 generated-corpus acceptance › rewrites a conversational comparison, expands its book collection, and performs a follow-up pass |
Drives the production static WebUI in Chromium and checks this outcome: rewrites a conversational comparison, expands its book collection, and performs a follow-up pass. | hybrid retrieval behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
Workspace RAG V2 generated-corpus acceptance › returns an explicit insufficient-evidence result instead of unrelated dense evidence |
Drives the production static WebUI in Chromium and checks this outcome: returns an explicit insufficient-evidence result instead of unrelated dense evidence. | hybrid retrieval behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
Workspace RAG V2 generated-corpus acceptance › returns immutable byte and line citations that reproduce the evidence |
Drives the production static WebUI in Chromium and checks this outcome: returns immutable byte and line citations that reproduce the evidence. | hybrid retrieval behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
Workspace RAG V2 generated-corpus acceptance › preserves multilingual source evidence while applying controlled translation |
Drives the production static WebUI in Chromium and checks this outcome: preserves multilingual source evidence while applying controlled translation. | hybrid retrieval behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
Workspace RAG V2 generated-corpus acceptance › enforces document, jurisdiction, and effective-date filters |
Drives the production static WebUI in Chromium and checks this outcome: enforces document, jurisdiction, and effective-date filters. | hybrid retrieval behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
Workspace RAG V2 generated-corpus acceptance › keeps cold content lexically searchable and lazily promotes only selected passages |
Drives the production static WebUI in Chromium and checks this outcome: keeps cold content lexically searchable and lazily promotes only selected passages. | hybrid retrieval behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
Workspace RAG V2 generated-corpus acceptance › bounds regular-expression search and rejects unsafe expressions |
Drives the production static WebUI in Chromium and checks this outcome: bounds regular-expression search and rejects unsafe expressions. | hybrid retrieval behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
Workspace RAG V2 generated-corpus acceptance › persists comparable metrics for every retrieval ablation |
Drives the production static WebUI in Chromium and checks this outcome: persists comparable metrics for every retrieval ablation. | hybrid retrieval behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
Workspace RAG V2 generated-corpus acceptance › diversifies duplicate evidence and preserves lexical retrieval after vector eviction |
Drives the production static WebUI in Chromium and checks this outcome: diversifies duplicate evidence and preserves lexical retrieval after vector eviction. | hybrid retrieval behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
Workspace RAG V2 generated-corpus acceptance › cancellation cannot replace an active immutable publication |
Drives the production static WebUI in Chromium and checks this outcome: cancellation cannot replace an active immutable publication. | hybrid retrieval behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
Workspace RAG V2 generated-corpus acceptance › publishes signature-isolated generations side by side |
Drives the production static WebUI in Chromium and checks this outcome: publishes signature-isolated generations side by side. | hybrid retrieval behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
Workspace RAG V2 generated-corpus acceptance › degrades safely when reranking is unavailable and isolates workspaces |
Drives the production static WebUI in Chromium and checks this outcome: degrades safely when reranking is unavailable and isolates workspaces. | hybrid retrieval behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
model label and expert selector use the same input meta typography |
Drives the production static WebUI in Chromium and checks this outcome: model label and expert selector use the same input meta typography. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
persists provider and font preferences across reloads |
Drives the production static WebUI in Chromium and checks this outcome: persists provider and font preferences across reloads. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
loads the shell, providers, conversations, files, plugins, and skills |
Drives the production static WebUI in Chromium and checks this outcome: loads the shell, providers, conversations, files, plugins, and skills. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
keeps only one Files, Architecture, Plugins, or Skills section expanded |
Drives the production static WebUI in Chromium and checks this outcome: keeps only one Files, Architecture, Plugins, or Skills section expanded. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
default file broker tool reads host files through the WebUI transport |
Drives the production static WebUI in Chromium and checks this outcome: default file broker tool reads host files through the WebUI transport. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
MISSING-13 architecture source and connector health tools work through the WebUI transport |
Drives the production static WebUI in Chromium and checks this outcome: architecture source and connector health tools work through the WebUI transport. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
governed architecture tools preserve records through the WebUI transport |
Drives the production static WebUI in Chromium and checks this outcome: governed architecture tools preserve records through the WebUI transport. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
architecture product panels expose sources, SQL approval, workflow approvals, graph evidence, and review cards |
Drives the production static WebUI in Chromium and checks this outcome: architecture product panels expose sources, SQL approval, workflow approvals, graph evidence, and review cards. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
MISSING-15 applies install-scoped branding consistently without unsafe HTML |
Drives the production static WebUI in Chromium and checks this outcome: applies install-scoped branding consistently without unsafe HTML. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
architecture sources render before the health report finishes |
Drives the production static WebUI in Chromium and checks this outcome: architecture sources render before the health report finishes. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
architecture SQL actions recover from transient approval and execution failures |
Drives the production static WebUI in Chromium and checks this outcome: architecture SQL actions recover from transient approval and execution failures. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
architecture graph retrieval selects the returned entity instead of stale list state |
Drives the production static WebUI in Chromium and checks this outcome: architecture graph retrieval selects the returned entity instead of stale list state. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
workflow rejection is idempotent under a rapid double click |
Drives the production static WebUI in Chromium and checks this outcome: workflow rejection is idempotent under a rapid double click. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
workflow operations center summarizes library, runs, approvals, and shadow outcomes |
Drives the production static WebUI in Chromium and checks this outcome: workflow operations center summarizes library, runs, approvals, and shadow outcomes. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
E2E-016 workflow compiler publishes a library entry and starts an approval-gated run |
Drives the production static WebUI in Chromium and checks this outcome: workflow compiler publishes a library entry and starts an approval-gated run. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
MISSING-08 workflow shadow lab records a human outcome and refreshes readiness metrics |
Drives the production static WebUI in Chromium and checks this outcome: workflow shadow lab records a human outcome and refreshes readiness metrics. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
workflow run inspector ignores stale responses after selecting another run |
Drives the production static WebUI in Chromium and checks this outcome: workflow run inspector ignores stale responses after selecting another run. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
workflow operations center keeps partial data usable and refreshes a failed service |
Drives the production static WebUI in Chromium and checks this outcome: workflow operations center keeps partial data usable and refreshes a failed service. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
MISSING-09 evaluation observability and ROI panel traces replay regressions and sponsor evidence |
Drives the production static WebUI in Chromium and checks this outcome: evaluation observability and ROI panel traces replay regressions and sponsor evidence. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
activates and deactivates compatible local plugins through the plugins panel |
Drives the production static WebUI in Chromium and checks this outcome: activates and deactivates compatible local plugins through the plugins panel. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
activates powershell plugin and invokes it through the WebUI transport |
Drives the production static WebUI in Chromium and checks this outcome: activates powershell plugin and invokes it through the WebUI transport. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
workspace selector lists, creates, renames, and switches workspaces |
Drives the production static WebUI in Chromium and checks this outcome: workspace selector lists, creates, renames, and switches workspaces. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
MISSING-03/MISSING-12 workspace RAG configuration panel saves paths and shows indexing progress |
Drives the production static WebUI in Chromium and checks this outcome: workspace RAG configuration panel saves paths and shows indexing progress. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
workspace settings discard unsaved edits when navigating to another conversation or dialog |
Drives the production static WebUI in Chromium and checks this outcome: workspace settings discard unsaved edits when navigating to another conversation or dialog. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
remembered facts persist across conversations and are used in later answers |
Drives the production static WebUI in Chromium and checks this outcome: remembered facts persist across conversations and are used in later answers. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
an immediate message after New waits for the new session |
Drives the production static WebUI in Chromium and checks this outcome: an immediate message after New waits for the new session. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
remembered facts are isolated between Cortex workspaces |
Drives the production static WebUI in Chromium and checks this outcome: remembered facts are isolated between Cortex workspaces. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
MISSING-11 memory browser command follows Inner voice and opens the in-page browser |
Drives the production static WebUI in Chromium and checks this outcome: memory browser command follows Inner voice and opens the in-page browser. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
MISSING-11 standalone memory browser loads, creates, edits, filters, and deletes through its own base URL |
Drives the production static WebUI in Chromium and checks this outcome: standalone memory browser loads, creates, edits, filters, and deletes through its own base URL. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
in-page memory browser can create, edit, search, and delete memories |
Drives the production static WebUI in Chromium and checks this outcome: in-page memory browser can create, edit, search, and delete memories. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
direct cognition tool calls can receive session and provider context |
Drives the production static WebUI in Chromium and checks this outcome: direct cognition tool calls can receive session and provider context. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
contextual search returns remembered facts with workspace RAG context |
Drives the production static WebUI in Chromium and checks this outcome: contextual search returns remembered facts with workspace RAG context. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
workspace RAG markdown context is automatically used during a conversation |
Drives the production static WebUI in Chromium and checks this outcome: workspace RAG markdown context is automatically used during a conversation. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
composer expert panel asks all experts and renders synthesis |
Drives the production static WebUI in Chromium and checks this outcome: composer expert panel asks all experts and renders synthesis. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
composer expert panel can run selected experts without synthesis |
Drives the production static WebUI in Chromium and checks this outcome: composer expert panel can run selected experts without synthesis. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
creates a conversation, sends a message, renders streaming output, tools, and usage |
Drives the production static WebUI in Chromium and checks this outcome: creates a conversation, sends a message, renders streaming output, tools, and usage. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
handles interactive prompt controls over the session event stream |
Drives the production static WebUI in Chromium and checks this outcome: handles interactive prompt controls over the session event stream. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
uploads and deletes workspace files through the files panel |
Drives the production static WebUI in Chromium and checks this outcome: uploads and deletes workspace files through the files panel. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
MISSING-05 plugin discovery returns stable compatible entries before runtime activation |
Drives the production static WebUI in Chromium and checks this outcome: plugin discovery returns stable compatible entries before runtime activation. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
attaches an uploaded workspace file to a message and prefers it over a matching RAG host path |
Drives the production static WebUI in Chromium and checks this outcome: attaches an uploaded workspace file to a message and prefers it over a matching RAG host path. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
opens skill editor, shows metadata and trigger controls, and saves |
Drives the production static WebUI in Chromium and checks this outcome: opens skill editor, shows metadata and trigger controls, and saves. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
skill deletion supports cancellation and confirmation |
Drives the production static WebUI in Chromium and checks this outcome: skill deletion supports cancellation and confirmation. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
renames, hides, and marks sessions via sidebar controls |
Drives the production static WebUI in Chromium and checks this outcome: renames, hides, and marks sessions via sidebar controls. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
shows stop control while a turn is busy and aborts the running turn |
Drives the production static WebUI in Chromium and checks this outcome: shows stop control while a turn is busy and aborts the running turn. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
mobile layout exposes the sidebar through the burger button |
Drives the production static WebUI in Chromium and checks this outcome: mobile layout exposes the sidebar through the burger button. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
mobile layout can navigate and operate architecture panels |
Drives the production static WebUI in Chromium and checks this outcome: mobile layout can navigate and operate architecture panels. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
mobile layout exposes workflow operations summary and run ledger |
Drives the production static WebUI in Chromium and checks this outcome: mobile layout exposes workflow operations summary and run ledger. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
mobile layout contains narrow-phone content and provides usable primary controls |
Drives the production static WebUI in Chromium and checks this outcome: mobile layout contains narrow-phone content and provides usable primary controls. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
mobile landscape keeps both memory browser panes usable |
Drives the production static WebUI in Chromium and checks this outcome: mobile landscape keeps both memory browser panes usable. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
E2E-001 workspace switch locks mutations and a failed switch restores the active workspace |
Drives the production static WebUI in Chromium and checks this outcome: workspace switch locks mutations and a failed switch restores the active workspace. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
E2E-002 sessions, files, skills, plugins, and memories remain isolated between workspaces |
Drives the production static WebUI in Chromium and checks this outcome: sessions, files, skills, plugins, and memories remain isolated between workspaces. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
E2E-003 stopping a slow turn ignores completion and allows a clean next turn |
Drives the production static WebUI in Chromium and checks this outcome: stopping a slow turn ignores completion and allows a clean next turn. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
E2E-004 editing governed SQL inputs invalidates approval and rapid execute is idempotent |
Drives the production static WebUI in Chromium and checks this outcome: editing governed SQL inputs invalidates approval and rapid execute is idempotent. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
E2E-006 workspace RAG save failure preserves edits and ignores stale workspace status |
Drives the production static WebUI in Chromium and checks this outcome: workspace RAG save failure preserves edits and ignores stale workspace status. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
E2E-007 failed plugin activation is atomic and core plugin removal is confirmation-gated |
Drives the production static WebUI in Chromium and checks this outcome: failed plugin activation is atomic and core plugin removal is confirmation-gated. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
E2E-009 workspace create and rename failures do not create phantom state |
Drives the production static WebUI in Chromium and checks this outcome: workspace create and rename failures do not create phantom state. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
E2E-010 file batch upload keeps successful files and reports individual failure |
Drives the production static WebUI in Chromium and checks this outcome: file batch upload keeps successful files and reports individual failure. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
E2E-011 memory browser surfaces version conflicts and permits a clean retry |
Drives the production static WebUI in Chromium and checks this outcome: memory browser surfaces version conflicts and permits a clean retry. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
E2E-012 skill editor degrades safely when the Markdown editor is offline |
Drives the production static WebUI in Chromium and checks this outcome: skill editor degrades safely when the Markdown editor is offline. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
E2E-013 expert panel blocks an empty explicit selection without a request |
Drives the production static WebUI in Chromium and checks this outcome: expert panel blocks an empty explicit selection without a request. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
E2E-014 source refresh failure keeps usable source data visible |
Drives the production static WebUI in Chromium and checks this outcome: source refresh failure keeps usable source data visible. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
E2E-017 failed regression suite is visibly release-blocking |
Drives the production static WebUI in Chromium and checks this outcome: failed regression suite is visibly release-blocking. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
E2E-018 context graph handles empty results and retries a transient retrieval failure |
Drives the production static WebUI in Chromium and checks this outcome: context graph handles empty results and retries a transient retrieval failure. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
E2E-019 review validation and creation failure preserve the form for retry |
Drives the production static WebUI in Chromium and checks this outcome: review validation and creation failure preserve the form for retry. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
E2E-021 missing sessions plugin shows the persistence warning |
Drives the production static WebUI in Chromium and checks this outcome: missing sessions plugin shows the persistence warning. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
E2E-022 font controls clamp at documented UI bounds and architecture tabs wrap by keyboard |
Drives the production static WebUI in Chromium and checks this outcome: font controls clamp at documented UI bounds and architecture tabs wrap by keyboard. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T2-E2E-002 routes normal turns through providers scoped to the active workspace |
Drives the production static WebUI in Chromium and checks this outcome: routes normal turns through providers scoped to the active workspace. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T2-E2E-003 ignores outgoing workspace events after a successful switch |
Drives the production static WebUI in Chromium and checks this outcome: ignores outgoing workspace events after a successful switch. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T2-E2E-004 deletes every disposable resource for only the named workspace |
Drives the production static WebUI in Chromium and checks this outcome: deletes every disposable resource for only the named workspace. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T2-E2E-005 renders workspace and session names as inert text |
Drives the production static WebUI in Chromium and checks this outcome: renders workspace and session names as inert text. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T2-E2E-007 restores an unaccepted prompt and retries it exactly once |
Drives the production static WebUI in Chromium and checks this outcome: restores an unaccepted prompt and retries it exactly once. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T2-E2E-008 keeps a file row and recovers when deletion fails |
Drives the production static WebUI in Chromium and checks this outcome: keeps a file row and recovers when deletion fails. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T2-E2E-010 round-trips and clears memory administrative fields |
Drives the production static WebUI in Chromium and checks this outcome: round-trips and clears memory administrative fields. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T2-E2E-011 preserves skill content when save fails and retries once |
Drives the production static WebUI in Chromium and checks this outcome: preserves skill content when save fails and retries once. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T2-E2E-012 retains successful expert evidence when one expert times out |
Drives the production static WebUI in Chromium and checks this outcome: retains successful expert evidence when one expert times out. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T2-E2E-013 keeps denied-source canaries out of graph retrieval |
Drives the production static WebUI in Chromium and checks this outcome: keeps denied-source canaries out of graph retrieval. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T2-E2E-014 ignores an older SQL plan that completes after a newer plan |
Drives the production static WebUI in Chromium and checks this outcome: ignores an older SQL plan that completes after a newer plan. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T2-E2E-015 presents a failed post-approval workflow action as failed |
Drives the production static WebUI in Chromium and checks this outcome: presents a failed post-approval workflow action as failed. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T2-E2E-016 renders finite honest ROI for zero cost and negative benefit |
Drives the production static WebUI in Chromium and checks this outcome: renders finite honest ROI for zero cost and negative benefit. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T2-E2E-017 normalizes graph terms and ignores an overlapping stale retrieval |
Drives the production static WebUI in Chromium and checks this outcome: normalizes graph terms and ignores an overlapping stale retrieval. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T2-E2E-018 keeps durable reviews isolated across workspaces |
Drives the production static WebUI in Chromium and checks this outcome: keeps durable reviews isolated across workspaces. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T2-E2E-019 rolls back failed optional-plugin removal and retries once |
Drives the production static WebUI in Chromium and checks this outcome: rolls back failed optional-plugin removal and retries once. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T2-E2E-021 restores focus when a destructive dialog is cancelled by keyboard |
Drives the production static WebUI in Chromium and checks this outcome: restores focus when a destructive dialog is cancelled by keyboard. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T2-E2E-022 reports a cross-tab memory edit conflict without overwriting |
Drives the production static WebUI in Chromium and checks this outcome: reports a cross-tab memory edit conflict without overwriting. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T2-E2E-023 explains a missing provider and retains the prompt |
Drives the production static WebUI in Chromium and checks this outcome: explains a missing provider and retains the prompt. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T3-E2E-001 sanitizes hostile Markdown, URLs, file names, and graph labels |
Drives the production static WebUI in Chromium and checks this outcome: sanitizes hostile Markdown, URLs, file names, and graph labels. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T3-E2E-002 fences a delayed file response to its initiating workspace |
Drives the production static WebUI in Chromium and checks this outcome: fences a delayed file response to its initiating workspace. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T3-E2E-006 excludes one denied-source canary from evidence product surfaces |
Drives the production static WebUI in Chromium and checks this outcome: excludes one denied-source canary from evidence product surfaces. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T3-E2E-008 preserves a valid provider through discovery failure and falls back deterministically |
Drives the production static WebUI in Chromium and checks this outcome: preserves a valid provider through discovery failure and falls back deterministically. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T3-E2E-009 reconciles one accepted turn after reload loses its stream |
Drives the production static WebUI in Chromium and checks this outcome: reconciles one accepted turn after reload loses its stream. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T3-E2E-010 handles empty and binary files, explicit collision choice, and failed-send attachment retention |
Drives the production static WebUI in Chromium and checks this outcome: handles empty and binary files, explicit collision choice, and failed-send attachment retention. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T3-E2E-012 paginates memories and recovers a stale delete without losing selection |
Drives the production static WebUI in Chromium and checks this outcome: paginates memories and recovers a stale delete without losing selection. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T3-E2E-013 requires an explicit decision before discarding unsaved skill content |
Drives the production static WebUI in Chromium and checks this outcome: requires an explicit decision before discarding unsaved skill content. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T3-E2E-014 retains independent expert evidence when synthesis alone fails |
Drives the production static WebUI in Chromium and checks this outcome: retains independent expert evidence when synthesis alone fails. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T3-E2E-016 keeps workflow modes and immutable versions distinct through multi-gate decisions |
Drives the production static WebUI in Chromium and checks this outcome: keeps workflow modes and immutable versions distinct through multi-gate decisions. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T3-E2E-017 reports replay reconstruction failure without executing writes |
Drives the production static WebUI in Chromium and checks this outcome: reports replay reconstruction failure without executing writes. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T3-E2E-018 enforces graph source constraints and drops malformed relationships |
Drives the production static WebUI in Chromium and checks this outcome: enforces graph source constraints and drops malformed relationships. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T3-E2E-019 validates review targets and deduplicates expert IDs before creation |
Drives the production static WebUI in Chromium and checks this outcome: validates review targets and deduplicates expert IDs before creation. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T3-E2E-020 persists optional plugin configuration across frontend reloads |
Drives the production static WebUI in Chromium and checks this outcome: persists optional plugin configuration across frontend reloads. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T3-E2E-022 traps focus and announces status for destructive workspace decisions |
Drives the production static WebUI in Chromium and checks this outcome: traps focus and announces status for destructive workspace decisions. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
T3-E2E-023 distinguishes provider discovery, missing plugin, and RAG diagnostics without destructive advice |
Drives the production static WebUI in Chromium and checks this outcome: distinguishes provider discovery, missing plugin, and RAG diagnostics without destructive advice. | browser-visible WebUI behavior and its transport/state boundaries; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
MISSING-11 standalone memory browser renders and reconciles cross-tab CAS conflicts |
Drives the production static WebUI in Chromium and checks this outcome: standalone memory browser renders and reconciles cross-tab CAS conflicts. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
MBS-2/MBS-3 standalone browser uses the injected workspace store when IndexedDB and OPFS are unavailable |
Drives the production static WebUI in Chromium and checks this outcome: standalone browser uses the injected workspace store when IndexedDB and OPFS are unavailable. | durable memory capture, recall, policy, browser, or cognition lifecycle; specifically the outcome and boundary stated in this title. | W: Live providers/token spend, production stores/settings, and live RAG databases; transport is the fake harness. | W: Chromium is installed, port 19787 is free, the serial harness contract is representative, and project-specific skips apply. |
Runner: npm run test:readme-qa. Titles in this section: 34.
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
README question set › [runtime-foundation] Which runtime is Cortex built on? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [projectmem-not-integrated] Is PROJECTMEM integrated into Cortex? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [default-workspace-name] What is the name of the single workspace a default Cortex installation provides? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [workspace-config-file] Which file inside a workspace defines its providers, active plugins, RAG, memory, skills, and the expert panel? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [local-services] Which local services does Cortex expose next to the Matbot process? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [userguide-document] Which document owns installation instructions and the WebUI tour? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [commands-document] Which document describes the purpose of every PowerShell script and the run.ps1 switches? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [memory-document] Which document explains remembered_facts, contextual_search, and memory-policy.json? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [reading-order] What reading order does the README recommend for the documentation? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [file-index-path] Which repository path holds the JSON-backed file index with keyword search and likely-secret skipping? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [file-broker-purpose] What is the file-broker module responsible for? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [mem0-stack-contents] Which components make up the Docker Compose stack under local-agent/docker/mem0? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [knowledge-directory] What does the local-agent/knowledge directory contain? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [technology-summary] Which languages and runtimes dominate Cortex overall? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [windows-service-wrapper] What wraps Matbot as a Windows service? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [file-index-glob-library] Which library does the file-index service use for glob exclusions? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [file-broker-dependencies] How many runtime dependencies does the file-broker service have? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [database-versions] Which Postgres and Neo4j versions does the Mem0 stack use? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [mem0-image-architecture] How does the Mem0 API image run on an amd64 host? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [node-version-requirement] Which minimum Node.js version does the Matbot runtime require? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [monorepo-package-manager] Which package manager does the Matbot runtime monorepo use? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [webui-client-stack] Which frontend framework does the Matbot WebUI client use? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [provider-adapter] What kind of provider adapter ships with Cortex and which endpoints does it work with? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [workspace-rag-hashing] Which hash does the workspace-rag plugin use for content hashing? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [workspace-rag-storage] Which persistent hybrid index backend does the workspace-rag plugin use? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [workspace-rag-cuda] What has to happen for workspace-rag to use CUDA embeddings? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [bge-reranking-plugin] Which retrieval plugin performs optional BGE reranking? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [retrieval-plugin-names] Which three retrieval plugins does the README list? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [expert-panel-plugin] What does the expert-panel plugin do? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [evaluation-observability-plugin] What does the evaluation-observability plugin provide? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [test-runners] Which test runners does the tests module use? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [no-provider-sdks] Does Cortex use vendor provider SDKs to reach models, and if not, what does it use? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [node-specific-packages] Where does Node-specific behavior live, given that shared Matbot packages avoid Node-only primitives? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
README question set › [no-web-framework-rationale] Why do the local HTTP services use Node's built-in http module instead of a web framework? |
Submits the README-grounded question through a real Cortex conversation and scores the answer 0/1. | documentation and executable contract consistency; specifically the outcome and boundary stated in this title. | Q: test:all, fake-harness behavior, and answer quality outside the README's binary regex rubric. |
Q: A real Cortex is running, README retrieval is available, a provider is configured, and token/time cost is acceptable. |
Runner: npm run test:integration:docker / npm run test:integration:cuda. Titles in this section: 3.
| Test title | What it does | What it covers | What it excludes | Assumptions |
|---|---|---|---|---|
MISSING-02/MISSING-14 disposable Docker stack validates health and credential-volume recreation |
Starts the generated disposable container fixture and checks this outcome: disposable Docker stack validates health and credential-volume recreation. | docker stack.integration.mjs behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | D: test:all, existing developer Compose projects/volumes, live providers, and browser behavior. |
D: Explicit Docker opt-in on a disposable host; CUDA cases also need NVIDIA runtime and their dedicated opt-ins. |
DCU-1/DCU-2/DCU-3/DCU-4/DCU-6/DCU-7 guarded CUDA sidecar validates hardware, parity, memory, performance, and cold E5 caching |
Starts the generated disposable container fixture and checks this outcome: guarded CUDA sidecar validates hardware, parity, memory, performance, and cold E5 caching. | docker stack.integration.mjs behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | D: test:all, existing developer Compose projects/volumes, live providers, and browser behavior. |
D: Explicit Docker opt-in on a disposable host; CUDA cases also need NVIDIA runtime and their dedicated opt-ins. |
WRS-4/WRS-5/WRS-8 V2 PostgreSQL keeps MiniLM and E5 dimension tables side by side |
Starts the generated disposable container fixture and checks this outcome: postgreSQL keeps MiniLM and E5 dimension tables side by side. | docker stack.integration.mjs behavior and the positive/negative boundary named by the test; specifically the outcome and boundary stated in this title. | D: test:all, existing developer Compose projects/volumes, live providers, and browser behavior. |
D: Explicit Docker opt-in on a disposable host; CUDA cases also need NVIDIA runtime and their dedicated opt-ins. |
| Test title | What it does and covers | Excludes | Assumptions |
|---|---|---|---|
| direct calls apply validation, policy, hooks and result redaction before returning content | Shared model/direct invocation, consent, policy loss and result hooks. | Live providers, external broker clients and a real browser. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| model and direct invocation share a rejecting tool hook | Shared model/direct invocation, consent, policy loss and result hooks. | Live providers, external broker clients and a real browser. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| an already cancelled direct request never calls the executor | Shared model/direct invocation, consent, policy loss and result hooks. | Live providers, external broker clients and a real browser. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| managed invocations fail closed without host policy and freeze nested rules | Shared model/direct invocation, consent, policy loss and result hooks. | Live providers, external broker clients and a real browser. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| file tool approval flags require runtime consent on both local and remote adapters | Shared model/direct invocation, consent, policy loss and result hooks. | Live providers, external broker clients and a real browser. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| Test title | What it does and covers | Excludes | Assumptions |
|---|---|---|---|
| contributions reject route collisions and release identity and lifetime on unload | Plugin ownership, configuration/vault durability, retrieval, deletion and adapter contracts. | Live databases, GPUs, crash recovery and external services. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| settings first writes serialize and configuration CAS preserves unrelated owner state | Plugin ownership, configuration/vault durability, retrieval, deletion and adapter contracts. | Live databases, GPUs, crash recovery and external services. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| configuration history redacts secrets and restores under the current schema and version | Plugin ownership, configuration/vault durability, retrieval, deletion and adapter contracts. | Live databases, GPUs, crash recovery and external services. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| federation distinguishes no matches from partial failure, isolates workspaces and follows unload | Plugin ownership, configuration/vault durability, retrieval, deletion and adapter contracts. | Live databases, GPUs, crash recovery and external services. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| workspace mutations reserve deletion and commit participants without an owned directory | Plugin ownership, configuration/vault durability, retrieval, deletion and adapter contracts. | Live databases, GPUs, crash recovery and external services. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| incomplete host-index discovery retains prior chunks and current policy still excludes denied paths | Plugin ownership, configuration/vault durability, retrieval, deletion and adapter contracts. | Live databases, GPUs, crash recovery and external services. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| provider model configuration preserves credentials and unrelated YAML and rejects stale writes | Plugin ownership, configuration/vault durability, retrieval, deletion and adapter contracts. | Live databases, GPUs, crash recovery and external services. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| vault serializes durable writes and failed persistence leaves memory unchanged | Plugin ownership, configuration/vault durability, retrieval, deletion and adapter contracts. | Live databases, GPUs, crash recovery and external services. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| profiles migrate hybrid selection without requiring legacy HTTP in the standard runtime | Plugin ownership, configuration/vault durability, retrieval, deletion and adapter contracts. | Live databases, GPUs, crash recovery and external services. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| expert definition refresh affects subsequent calls while in-flight panels retain their snapshot | Plugin ownership, configuration/vault durability, retrieval, deletion and adapter contracts. | Live databases, GPUs, crash recovery and external services. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| loader rollback removes failed contributions; unloading an older provider preserves its replacement | Plugin ownership, configuration/vault durability, retrieval, deletion and adapter contracts. | Live databases, GPUs, crash recovery and external services. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| persistent BGE memory contributes to federation without taking over KnowledgeIndex | Plugin ownership, configuration/vault durability, retrieval, deletion and adapter contracts. | Live databases, GPUs, crash recovery and external services. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| Test title | What it does and covers | Excludes | Assumptions |
|---|---|---|---|
| standard profile starts with in-process file services, configuration and contributed feature UI | Isolated real Node startup profiles and authorized HTTP capability adapters. | The user workspace, Windows service installation and Docker. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| minimal profile exposes no implicit administration or feature panels | Isolated real Node startup profiles and authorized HTTP capability adapters. | The user workspace, Windows service installation and Docker. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| workspace and diagnostic HTTP adapters enforce the shared host permission policy | Isolated real Node startup profiles and authorized HTTP capability adapters. | The user workspace, Windows service installation and Docker. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| Test title | What it does and covers | Excludes | Assumptions |
|---|---|---|---|
| unloading a feature removes its markup and listeners and reloading installs one copy | Feature lifecycle, minimal chat metadata, contributed panels and assembled browser startup. | Live providers, external CDN availability and production deployment. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| chat shell starts with an empty capability list | Feature lifecycle, minimal chat metadata, contributed panels and assembled browser startup. | Live providers, external CDN availability and production deployment. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| additional configuration and diagnostics views mount without editing the shell | Feature lifecycle, minimal chat metadata, contributed panels and assembled browser startup. | Live providers, external CDN availability and production deployment. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |
| assembled browser runtime boots the shared shell with plugin-owned administration | Feature lifecycle, minimal chat metadata, contributed panels and assembled browser startup. | Live providers, external CDN availability and production deployment. | Node 24+, local fixtures; browser cases require installed Playwright Chromium. |