From e71a44776194ac1b03e73f831156241bfa1d222c Mon Sep 17 00:00:00 2001 From: gethin Date: Sun, 12 Jul 2026 13:15:48 +0100 Subject: [PATCH 1/4] docs: e2e clean-slate smoke-test design (agent-team phase 2 synthesis) Design for three-tier e2e smoke suite: T1 hermetic sqlite, T2 hermetic agentcore via AWS_ENDPOINT_URL fake, T3 gated live AWS. Produced by multi-agent workflow: 6 empirical verifiers, 5 scenario authors, 2 adversarial judges per set (60 keep / 40 fix / 7 kill), final synthesis. Documents 12 product defects pinned as current behavior. Co-Authored-By: Claude Fable 5 --- ...2026-07-12-e2e-clean-slate-smoke-design.md | 207 ++++++++++++++++++ 1 file changed, 207 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-12-e2e-clean-slate-smoke-design.md diff --git a/docs/superpowers/specs/2026-07-12-e2e-clean-slate-smoke-design.md b/docs/superpowers/specs/2026-07-12-e2e-clean-slate-smoke-design.md new file mode 100644 index 0000000..c392b34 --- /dev/null +++ b/docs/superpowers/specs/2026-07-12-e2e-clean-slate-smoke-design.md @@ -0,0 +1,207 @@ +# better-memory E2E Smoke-Test Design — Final (Phase 2 Synthesis) + +Goal: prove a **brand-new user** (no `~/.better-memory`, no MCP registration, no hooks, no DB) can set up and use better-memory in both sqlite and agentcore modes — without any test ever touching real user data, real AWS, or the network. + +Tiers: **T1** hermetic sqlite (default run) · **T2** hermetic agentcore via `AWS_ENDPOINT_URL` fake (default run) · **T3** live AWS (gated: `integration` marker + `BETTER_MEMORY_TEST_AGENTCORE=1`). + +All scenario edits from the judge round are applied; scenarios killed by both judges are dropped; single-judge kills are folded/trimmed per the surviving judge's critique; all judge-proposed gap scenarios are incorporated; cross-lens overlaps are deduplicated to the stronger variant. + +--- + +## 1. Final Scenario Catalog + +### A. Installer surface (`install_hooks`) — T1 + +| id | tier | mode | purpose | regression caught | +|---|---|---|---|---| +| `e2e-install-fresh-clean-slate` | T1 | both | Truly clean slate (no `.claude/`, no `.claude.json`): subprocess install writes both targets with exactly 5 hook entries, quoted interpreters (paths **with embedded space**), correct py/pyw split, matchers, async flags, `mcpServers` env block; zero DB/side writes; symlinks capability-probed. | needs_stdout interpreter swap (pythonw silently kills additionalContext); dropped interpreter quoting; missing UserPromptSubmit/PreToolUse merge branches; installer growing a DB-creating side effect. | +| `e2e-install-idempotent-rerun` | T1 | both | Runs 2 and 3 are **byte-identical** for both targets; better-memory entry count stays exactly 5. | REMOVE-pass predicate drift (exact-match vs substring) causing duplicate hook groups appended per re-run; serialization drift. | +| `e2e-install-foreign-config-preserved` | T1 | both | Seeded foreign servers/hooks/top-level keys + user's custom `BETTER_MEMORY_HOME` survive; command path refreshed; **legacy `session_start`/`session_retrieve` entries scrubbed** while co-resident user hooks in the same group survive (legacy seeds folded in per judges — no standalone legacy scenario). | `env.setdefault` → unconditional assignment (repoints user data dir); dict rebuild dropping `model`/foreign events; `_LEGACY_HOOK_MODULES` deletion; group-level (vs entry-level) removal. | +| `e2e-install-malformed-claude-json-refused` | T1 | both | Malformed `~/.claude.json` (first target — direction existing tests don't cover): exit 1, path+lineno+'Fix the file then re-run' on stderr, **both** files byte-unchanged, no backups, no `*.tmp` litter. (Reframed per judge: the per-target-loop ordering trap is owned by the existing malformed-settings test; this pins the swallow-and-treat-as-`{}` hazard for `.claude.json`, whose loss would destroy the user's whole MCP config.) | catch-JSONDecodeError-continue-with-`{}` change atomically replacing the user's `.claude.json`. | +| `e2e-install-backup-before-overwrite` | T1 | both | Backups land in `$BETTER_MEMORY_HOME/install-backups/` with timestamped names containing the **pre-run bytes**; dir auto-created. | `_backup` call moved after `_atomic_write` (backup-of-result destroys the only rollback artifact); backup location/name-format drift docs point users at. | +| `e2e-install-symlink-oserror-fallback` | T1 | both | Driver patches `Path.symlink_to` to raise `OSError(1314)`: exit 0, ≥2 `WARN skill symlink skipped` + `Developer Mode` on stderr, both configs still fully written. | Narrowing/removing the `except OSError` — total install failure for non-Developer-Mode Windows (symlinks run *before* JSON writes). | +| `e2e-install-symlink-replacement-ladder` (gap) | T1 | both | Capability-probed: pre-existing wrong symlink / plain file / real dir with sentinel at the skill path each get replaced by a correct symlink; sentinel gone (pins the documented no-backup destructive contract); no WARN; already-correct link is a no-op (lstat compare). | Removing pre-clearing → `FileExistsError` (an OSError) silently converts every upgrade into perpetual "WARN skipped" — skills never update again. | + +### B. `setup.sh` surface — T1 (skipif bash absent; containment per harness gap-1) + +| id | tier | mode | purpose | regression caught | +|---|---|---|---|---| +| `e2e-setup-headless-decline-completes` | T1 | both | `printf 'n\n' \| bash scripts/setup.sh` runs to completion: layout dirs (`spool/`, `knowledge-base/{standards,languages,projects}`), install_hooks invoked, `win_path`-correct command in `.claude.json` (regex on decoded backslash/drive form), py/pyw split survives bash→argparse. **Fixes applied:** fake-`curl`-exits-1 shim (+ no-op `ollama`) forces the daemon-unreachable warn-and-continue branch on every host — no network pull ever; env carries SYSTEMROOT/COMSPEC/TEMP/TMP/UV_CACHE_DIR(or LOCALAPPDATA); bash-version probe for patsub_replacement backslash semantics; assert `Dependencies installed.` before `Runtime layout ready`. | `win_path` regressions (MSYS `/c/...` handed to Claude Code); mkdir-layout/install stage removal or reordering; `set -e`-unsafe edit to the decline branch. | +| `e2e-setup-eof-aborts-all-or-nothing` | T1 | both | `bash scripts/setup.sh < /dev/null`: exit 1 at the Ollama `read -rp`, **before** layout and install_hooks — no `.claude.json`, no settings, no spool. Skip-guard probes both ollama-detection paths (self-skips on this dev machine; runs on ollama-free CI — documented). Asserts `Dependencies installed.` + `Ollama not found` so an earlier uv-stage death cannot false-pass. | "Do the important stuff first" reordering → headless failure leaves hooks registered against a setup the user believes failed; prompt gaining an EOF-safe default (deliberate contract flip). | +| `e2e-setup-default-home-derivation` (gap) | T1 | both | Decline flow with `BETTER_MEMORY_HOME` **absent**: default `$HOME/.better-memory` derived, propagated via `--home` into `mcpServers` env and backup location. | Dropping the `${VAR:-default}` fallback / default dir rename / wrong `--home` plumbing — invisible while every other test exports the var. | +| `e2e-setup-install-hooks-failure-propagates` (gap) | T1 | both | Malformed seeded `.claude.json` + decline flow: exit 1, `install_hooks failed` + `aborting` + underlying `:` in output, no final `Done.`; `spool/` EXISTS (layout precedes install — stage-order pin); settings.json absent, `.claude.json` bytes unchanged. | `\|\| true` / dropped subshell status masking an install failure as "Done." | + +Containment (harness gap-1, embedded): setup.sh tests run against a **copy of the repo** (or `UV_PROJECT_ENVIRONMENT`+`UV_CACHE_DIR` redirect) with a warm host uv cache; assert the real repo `.venv/pyvenv.cfg` hash unchanged and all writes landed under tmp. + +### C. sqlite journey — T1 + +| id | tier | mode | purpose | regression caught | +|---|---|---|---|---| +| `e2e-sqlite-first-boot-migrates-tools-knowledge` | T1 | sqlite | First `python -m better_memory.mcp` on a **nonexistent** home: initialize succeeds offline (poisoned OLLAMA_HOST + `EMBEDDINGS_BACKEND=sqlite`), full tool subset incl. both synthesize tools, both DBs created+migrated (table superset incl. trigram FTS), `knowledge-base/` NOT auto-created (defect pin). **Folded gap-3:** `knowledge.search` + `knowledge.list` return well-formed empty results, not errors, on the unindexed corpus. | Clean-slate-breaking migration; `apply_migrations` moved out of `create_server`; `supports_synthesis` gating regression; knowledge handlers raising on empty index (breaks every fresh install's mandated startup `knowledge_search`). | +| `e2e-sqlite-hook-before-server-degraded` (merged set1+set3) | T1 | sqlite | KNOWN-DEFECT PIN: SessionStart hook on a virgin home → exit 0, empty stderr, single fallback-directive envelope (prefix-match `OperationalError: no such table:`), schema-less `memory.db` (0 tables), **no** `runtime/sessions` marker, no other dirs, silent `hook_errors` no-op. | Flips red on the intended fix (hook-side migrations / hoisted marker write) forcing a deliberate contract update; never-fail wrapper regressions that would break every new user's session start. | +| `e2e-sqlite-hook-first-then-server-heals` (gap-1; absorbs marker-bridge scenario) | T1 | sqlite | The real new-user **sequence**: degraded hook → server boot onto the pre-existing 0-table WAL file heals it (migrations apply) → `memory.session_bootstrap` MCP tool works (`episode.action == 'opened'`) → re-run hook: `Episode: reused`, marker written with content `e2e-session-1`, exactly 1 open episode row, full table superset. | Migrations failing on an already-existing empty DB; fallback directive pointing at a drifted tool name; episode-dedup and marker-bridge regressions (folded from the split-verdict marker-bridge scenario). | +| `e2e-sqlite-observe-retrieve-record-use-offline` (absorbs record_use scenario per both judges) | T1 | sqlite | Full offline data loop in one server session: observe → trigram retrieve (obs_id surfaces) → bucket shape `{do,dont,neutral}` → `record_use(success)` on obs A **and** `record_use(failure)` on obs B → not-found id returns isError → post-exit re-open DB: A has used_count=1/validated_true=1/score≈+1.0, B has validated_false=1/score≈−1.0, timestamps non-NULL. Observe wall time < 30s. | sqlite-embeddings bypass loss (Ollama path reintroduced → hangs/EmbeddingError for every Ollama-less user); migration-0011 trigger drop; success/failure branch swap (failure leg per judge); dropped commit losing ratings at process exit (post-exit read). | +| `e2e-sqlite-session-close-rate-then-marker` | T1 | sqlite | Two-fire Stop sequence with a real rating turn: fire 1 blocks with RATE_MEMORIES (memory id + `rate-session-memories` in directive), **zero** session_end markers; server spawned with `cwd=proj_dir` (or `CLAUDE_PROJECT_DIR` on both sides) and `CLAUDE_SESSION_ID` deleted → session resolved via marker file; ratings payload exactly `[{'kind','id','class':'ignored'}]`; assert `sum(result['applied'].values()) >= 1`; exposures asserted on real wire shape (no `rated_at` field); fire 2: **empty stdout**, exactly one session_end marker. | Marker-written-while-blocking (double session_end / premature synthesis); unrated-dedupe query silently matching nothing; `rated_at` never stamped → infinite Stop-block loop; marker-bridge break in `apply_session_ratings`. | +| `e2e-sqlite-spool-drain-synthesize-loop` | T1 | sqlite | session_end marker + next-session `memory.retrieve` drains spool (top-level non-recursive glob), closes background episode as `no_outcome`; `start_episode` payload `pending_synthesis['pending'] >= 1`; get_context carries obs_id; apply asserts `payload['ok'] is True`, `payload['counts']['created'] == 1`, `queue` keys present; second get_context excludes E; final retrieve surfaces the reflection in `dont`. Session B env carries `CLAUDE_SESSION_ID=e2e-session-2` explicitly. | Drain no longer closing episodes (learning loop strands forever); queue-count drift zeroing the CLAUDE.md synthesize trigger; lost already-synthesized guard duplicating reflections; `pending_synthesis` field dropped from serializer. | +| `e2e-sqlite-contextual-inject-contract` (gap-2 set 1) | T1 | sqlite | (a) virgin home + **default mode** (var unset = 'both'): exit 0, never a traceback; (b) migrated home + seeded matching memory: injection envelope + `session_memory_exposure` row `source='contextual'`; (c) `mode=off`: exit 0, exact empty envelope, `BETTER_MEMORY_HOME` **not created** (verified zero side effects). | Never-fail wrapper regression breaking *every prompt* for every new user; off-mode gaining side effects; injection/exposure recording break. | +| `e2e-sqlite-ollama-absent-default-backend` | T1 | sqlite | Default (ollama) embeddings, `OLLAMA_HOST=127.0.0.1:9`: boot succeeds with stderr warning; `memory.observe` → in-band isError EmbeddingError (timed **only around the call**, budget **< 30s**, measured baseline 7.7s noted); `knowledge.list` still works; KNOWN-DEFECT PIN: orphan episode row committed (episodes=1, observations=0). | Fatal startup probe (bricks Ollama-less installs); EmbeddingError escaping as server crash; retry/timeout inflation (~90s stalls); silent trigram auto-fallback (deliberate change flag); orphan-episode fix lands. | + +### D. agentcore hermetic — T2 (fake `AWS_ENDPOINT_URL` endpoint; verified boto3 1.43.14 seam) + +| id | tier | mode | purpose | regression caught | +|---|---|---|---|---| +| `e2e-ac-server-boot-tools-hidden` | T2 | agentcore | Real server boots from fabricated `agentcore.json` against the fake endpoint; synthesize tools hidden; sqlite DBs still migrated (docs-contradiction pin). Fake routing derived from **gzipped** `service-2.json.gz` botocore models (judge fix). | `supports_synthesis` flip re-exposing no-op tools; broken factory agentcore branch; boot growing an AWS call. | +| `e2e-ac-retrieve-polarity-fanout` | T2 | agentcore | The **flagship wired-path** test: MCP `memory.retrieve` → exactly 3 `ListMemoryRecords` (episodic memory, `reflections` namespace), order-insensitive polarity set `{do,dont,neutral}` + `status=active` filter each; buckets are exact empty lists; optional single-polarity leg (`polarity='do'` → exactly 1 request). Semantic-search leg **deleted** (unwired at MCP layer, per both judges). | Fan-out collapsed into one unfiltered list; dropped status filter resurfacing retired reflections; ReflectionToolHandlers rewired off the backend. | +| `e2e-ac-mcp-dispatch-gap-pin` (new, per judges) | T2 | agentcore | KNOWN-DEFECT PIN of the **dispatch gap**: in agentcore mode, `memory.observe` / `memory.semantic_observe` / `memory.record_use` produce **zero** wire requests and read/write local `memory.db` (row lands in sqlite; observe returns a local uuid). Docstring cites `server.py:250`; delete when handlers are wired to the backend. | Flips loudly the day dispatch wiring lands, forcing wire tests to be promoted; guards against anyone claiming AWS coverage from these tools. | +| `e2e-ac-backend-wire-fidelity` (re-homed, per judges) | T2 | agentcore | In-process `AgentCoreBackend` with **real boto3 clients** against the fake (no mocks — real botocore serialization): CreateEvent shape (EPI memory routing, sessionId/actorId, USER role, stringValue-only metadata, None-dropped keys, dummy env IDs never on the wire); semantic BatchCreate (`requestIdentifier == sha256(content)[:80]` computed independently, SEM routing, strategy id, initial counters); `record_use` + `credit_one` full-snapshot updates with `x-amz-agentcore-memory-*` keys **stripped** (currently untested anywhere; live-AWS-400 class); observe with no session id → ValueError, zero wire requests (set-3 gap-2 folded here). | Dedup-identifier scheme change; semantic/episodic routing swap; reserved-prefix strip removal (real-AWS 400); env-ID consumption (would put dummies on the wire); uuid4 session fallback fabricating identities. | +| `e2e-ac-session-close-closure-and-env-gate` (merged + set-3 gap-1) | T2 | agentcore | Stop hook fires exactly one role=OTHER closure CreateEvent (real client construction — zero coverage today), SigV4 region == **agentcore.json's** region, spool marker still written; Case B (`BETTER_MEMORY_STORAGE_BACKEND` absent — exactly what the installer produces): exit 0, zero wire requests, marker **still written**, no `hook_errors` row (guard short-circuits pre-try). | Closure reorder/removal (silent AWS extraction latency loss); boto exception killing the marker; env-propagation fix landing (deliberate flip); guard condition change. | +| `e2e-ac-contextual-inject-wire-and-degradation` (merged set-2 gap-1 + set-3 inject) | T2 | agentcore | The only shipped per-prompt backend path: Case A happy — hook subprocess hits SEM/EPI on the fake, envelope contains injection; Case B misconfig (ID vars unset), clean slate — exit 0, empty envelope, zero wire, **stray schema-less `memory.db` created** (defect pin), no state/; Case C misconfig, pre-migrated DB — exactly one `hook_errors` row (`hook_name='contextual_inject'`, `exception_type='ValueError'`, column **`exception_message`** per migration 0005 — judge fix) containing both var names. | Per-prompt never-fail regression; SeenStore hoisted above `get_config` (state litter); gate removal flipping Case B/C (rewrite alongside fix); silent-degradation contract drift. | +| `e2e-ac-region-split-brain-pin` (merged set-2 gap-2 + set-3) | T2 | agentcore | KNOWN-DEFECT PIN: `agentcore.json` region=`us-east-1`, env region unset → (a) server `memory.retrieve` (the **wired** trigger — corrected from observe per dispatch-gap finding) signs SigV4 scope `eu-west-2` (env default) with `MEM-EPI-JSON` in path and `DUMMY-EPI` absent; (b) session_close hook signs `us-east-1` (json-derived). One test proves the two planes disagree. | Single-source-of-truth fix flips both halves visibly; factory consuming dummy env IDs; hardcoded endpoint_url breaking the env seam. | +| `e2e-ac-neg-prehandshake-config-errors` (parametrized: idvar-gate + backend typo; absorbs set-3 typo scenario per judge) | T2 | agentcore | Raw spawn, two cases sharing one helper: (1) KNOWN-DEFECT idvar gate — valid `agentcore.json`, ID vars unset → rc 1, stdout `''` (pre-handshake), ValueError text + unsatisfiable `agentcore init` remediation; docstring: delete with the dummy-var fixture when gate is fixed. (2) `BETTER_MEMORY_STORAGE_BACKEND=AgentCore` typo → rc 1, stdout `''`, `is not one of` + repr'd `'AgentCore'`. Both cases: **no memory.db created** (config validated before any disk write). | Gate fix/removal (inverted trap forcing workaround cleanup); silent sqlite fallback/coercion on invalid values; `get_config` moved past the handshake. | +| `e2e-ac-neg-missing-agentcore-json` (absorbs the SDK-client-experience scenario) | T2 | agentcore | The **enduring** negative (survives the gate fix), two levels: raw — rc 1, stdout `''`, `agentcore.json not found` + `better-memory agentcore init`, and `memory.db` migrated-before-backend-failure (sqlite_master > 0; ordering pin + docs-claim pin); SDK — `pytest.raises(McpError)` tight around `initialize()`, `error.code == -32000`, `Connection closed`, error text recovered via real-file `errlog`; contexts exit cleanly; never assert exit code via SDK. | Silent sqlite fallback on missing json (worst case); remediation text loss; half-alive server advertising tools with no backend; SDK upgrade changing the client-visible death shape. | +| `e2e-ac-neg-corrupt-agentcore-json` | T2 | agentcore | Truncated JSON and `schema_version: 2` both → rc 1, stdout `''`, `AgentCoreConfigError` naming the file; `failed to parse` / `unsupported schema_version=2; expected 1`; negative pin: **no** `agentcore init` remediation (product gap flag). | `load_agentcore_config` swallowing parse errors into None (fail-loud contract destroyed); unshipped schema bumps. | +| `e2e-ac-neg-boto3-missing` | T2 | agentcore | PYTHONPATH-shadow `boto3.py` raising ModuleNotFoundError: rc 1, stdout `''`, raw traceback, PRODUCT-GAP PIN: no `better-memory[agentcore]` / `pip install` hint (inverted assertion comment). Control run assertion is **content-based** (`ModuleNotFoundError` absent from control stderr — judge fix; never rc comparison). | Friendly-message fix landing (flip); boto3 becoming a top-level import breaking sqlite-mode plain installs; lazy-import contract break. | + +### E. live AWS — T3 (integration marker + `BETTER_MEMORY_TEST_AGENTCORE=1`) + +| id | tier | mode | purpose | regression caught | +|---|---|---|---|---| +| `e2e-live-init-status-journey` | T3 | agentcore | `agentcore init` (monkeypatched `bm_int_*` DEFAULT names, cleanup pre-registered) provisions two ACTIVE memories, writes schema-1 `agentcore.json` with distinct ids + non-empty strategy ids; re-init refuses (rc 1, `already exists`/`force`, file unchanged — money-safety pin); `status` subprocess reports both ids + ACTIVE. | Real-AWS strategy/indexedKeys shape drift; poll loop returning pre-ACTIVE; init writing json before confirmation; silent-clobber orphaning billable memories. | +| `e2e-live-smoke-retrieve-backend-roundtrip` (rebuilt per both judges) | T3 | agentcore | (a) `agentcore smoke` CLI passes against session-scoped throwaway memories (6-step data-plane loop vs real AWS wire validation — mocked-only today); (b) real MCP server boot with real creds + **explicit** `BETTER_MEMORY_AGENTCORE_REGION` (split-brain workaround, commented) → `memory.retrieve` live (wired path; empty `{do,dont,neutral}` — live filter/namespace validation); (c) direct `AgentCoreBackend.observe` → `list_observations` with unique marker: read-after-write + metadata survival (`outcome`/`theme` through the flattening). MCP observe round-trip **removed** (tautological-sqlite false positive per both judges). | AWS-side payload/enum/metadata validation drift the fakes structurally cannot see; ListEvents mapping regressions; live metadataFilters rejection. | + +### F. Harness safety & suite meaningfulness (tests/e2e_meta + fixtures + scripts) + +| id | tier | mode | purpose | regression caught | +|---|---|---|---|---| +| `meta-canary-home-run` | meta | both | Whole e2e suite executed under a seeded hostile canary home (invalid-sqlite `memory.db` sentinel); outer env strips all `BETTER_MEMORY_*/CLAUDE_*/AWS_*/OLLAMA_*` **case-insensitively** (judge fix) so an un-redirected fixture falls back into the canary; sha256 + file-set diff + mcpServers subset unchanged; explicit exit-5 vacuity assertion. | Any fixture forgetting USERPROFILE/HOME/BETTER_MEMORY_HOME — install_hooks would corrupt real `~/.claude.json` (no `--target` flag exists). | +| `real_home_canary` fixture | meta | both | Session-scoped autouse tripwire on every run: additive dot-canaries (xdist worker-id suffixed, stale-overwrite on setup), **semantic subtree comparison** of the better-memory mcpServers/hooks entries (not byte hash — live-session-safe, per judges), tmp-path smoking-gun check, install-backups/skills-set tell-tales; hash mismatch demoted to warning unless corroborated; env snapshot at module-import time covering both session-id var names. | A single new test with a hand-rolled env dict — immediate local failure instead of corrupted personal config days later. | +| `meta-env-bleed-poisoned-shell` | meta | both | Re-run T1 slice under a maximally hostile outer shell (backend=agentcore, POISON ids, decoy home, aggressive inject, poisoned AWS/Ollama): inner suite still green, decoy untouched, no poison strings in output. | Reintroduction of `{**os.environ, ...}` spawn patterns; unpinned `BETTER_MEMORY_PROJECT`/backend leaking into outcomes. | +| `e2e-ollama-zero-traffic-tripwire` | T1 | sqlite | Recorder server on OLLAMA_HOST across the full journey + both sync hooks: **zero requests**, with positive journey assertions (anti-vacuity). | Ollama probe/embed path becoming unconditional again — stronger than the `.invalid` poison (proves no attempt, not just survivable failure). | +| `e2e-aws-lockdown-tripwire` | T2 | agentcore | Helper-contract + round-trip: every request Host `127.0.0.1:*`, every SigV4 scope `Credential=bm-e2e-fake/`, credentials-file lockdown, IMDS disabled; dummy MEMORY_ID vars grep-pinned to the single helper + the one defect-pin test. | Dropping `AWS_ENDPOINT_URL` (real endpoint hit) or credential lockdown (real AKIA key signed) — red **before** anything is billed; workaround metastasis. | +| `meta-marker-tier-wiring` | meta | both | Default collection = all T1+T2, zero T3; `e2e` marker declared (no unknown-mark warning); `-m integration` without the env gate → green-**skipped**, never red/exit-5. | Hermetic test mismarked `integration` (silent coverage rot); marker declaration loss; skip-gate refactored to assert. | +| `meta-env-helper-contract` | meta | both | `isolated_env()` contract: HOME+USERPROFILE both set on every OS, case-insensitive key hygiene, SYSTEMROOT/COMSPEC/TEMP/TMP allowlist, child-process ground truth (`Path.home()`/`expanduser` == tmp in a real spawn); AST/grep check that every spawn site uses the helper; setup.sh/symlink skip-decoration checks. | POSIX contributor deleting the "redundant" USERPROFILE line → every Windows run torches the real profile; helper bypass via hand-rolled env dicts. | +| mutation smoke (M1–M4) | meta | both | See §5. Runs in a throwaway **git worktree** (judge fix: current tree has ~70 untracked scratch files; gate on `--untracked-files=no`); expected-node-id matching; one-other-test-passes; unpatched control run. | The suite itself decaying into a green-always ritual — including the safety harness (M4). | + +Dropped (both-judge or fold-resolved): `t1-observer-hook-spools-tool-event` (redundant with `tests/hooks/test_observer.py`; three one-line additions go there instead — see task 11), `t1-install-legacy-entry-migration` (folded into foreign-config), `t1-bootstrap-first-opened-second-reused` (folded into heal sequence), MCP-level agentcore observe/semantic/record_use wire scenarios (false premise — re-homed to backend wire-fidelity + dispatch-gap pin), `neg-agentcore-observe-no-session-id` at MCP level (unreachable via MCP today — folded into backend wire-fidelity), MCP semantic_retrieve search leg (schema rejects `search`; unwired). + +--- + +## 2. Harness Architecture + +``` +tests/e2e/ + conftest.py # clean_slate_home, real_home_canary (autouse, session), mcp_session, run_hook + _env.py # isolated_env(tmp_home, **pins) — THE single env choke point + _agentcore_env.py # agentcore_env(tmp, port) — extends isolated_env; ONLY location of dummy ID vars (FIXME) + _fake_agentcore.py # ThreadingHTTPServer fake; routing from botocore service-2.json.gz (gzip-decoded) + test_install_hooks.py # scenarios A1-A7 + test_setup_sh.py # scenarios B1-B4 (skipif no bash; repo-copy containment) + test_sqlite_journey.py # C1-C6 + test_hooks_contracts.py# C2, C7 (inject), degraded/heal + test_sqlite_negative.py# C8 (ollama-absent) + test_agentcore_t2.py # D1-D7 + test_agentcore_neg.py # D8-D11 + test_tripwires.py # ollama zero-traffic, aws lockdown +tests/e2e_meta/ + test_canary_home.py # meta-run + test_env_bleed.py + test_marker_wiring.py + test_env_helper_contract.py + mutations/M1..M4.patch +tests/integration/ + test_agentcore_live_e2e.py # T3 (integration marker; reuses conftest throwaway memories) +scripts/e2e_mutation_smoke.py +``` + +**Key fixtures / helpers** + +- `isolated_env(tmp_home, **pins)` — built from an **allowlist**, never `os.environ.copy()`: sets `HOME`, `USERPROFILE` (both, unconditionally, every OS), `HOMEDRIVE`/`HOMEPATH` (win), `BETTER_MEMORY_HOME=/.better-memory`, `BETTER_MEMORY_PROJECT`, `CLAUDE_SESSION_ID`, `BETTER_MEMORY_EMBEDDINGS_BACKEND=sqlite`, `OLLAMA_HOST=http://does-not-exist.invalid:1`; preserves only `{PATH, SYSTEMROOT, COMSPEC, PATHEXT, TEMP, TMP, WINDIR, LANG, LC_ALL, PYTHONIOENCODING}`; drops all `CLAUDE_*/BETTER_MEMORY_*/AWS_*/OLLAMA_*` **case-insensitively**. +- `agentcore_env(tmp, fake_port)` — adds `BETTER_MEMORY_STORAGE_BACKEND=agentcore`, `BETTER_MEMORY_AGENTCORE_REGION`, `AWS_ENDPOINT_URL=http://127.0.0.1:`, `AWS_ACCESS_KEY_ID/SECRET=bm-e2e-fake*`, `AWS_SHARED_CREDENTIALS_FILE`/`AWS_CONFIG_FILE` → nonexistent tmp paths, `AWS_EC2_METADATA_DISABLED=true`, removes `AWS_IGNORE_CONFIGURED_ENDPOINT_URLS`, and the two dummy `BETTER_MEMORY_AGENTCORE_*_MEMORY_ID` vars **in this one place only**, with a `# FIXME(idvar-gate)` comment: workaround for the dead gate at `config.py:293-301`; delete together with `e2e-ac-neg-prehandshake-config-errors[idvar]`. +- `fake_agentcore_endpoint` — local ThreadingHTTPServer, ephemeral 127.0.0.1 port bound before env construction; records `(method, path, headers, body)`; canned rest-json responses per operation; routing derived from the installed `botocore/data/bedrock-agentcore*/**/service-2.json.gz` (gzip-decoded). Plain HTTP; verified end-to-end at boto3 1.43.14. +- `mcp_session(params_env, errlog=None)` — `stdio_client` + `ClientSession(read_timeout_seconds=60)`; **always** includes `USERPROFILE`/`HOME` explicitly in `StdioServerParameters.env` (the SDK force-inherits them on Windows underneath `server.env`); `errlog` must be a real file with `fileno()` (never StringIO); supports `cwd=` for marker-bridge tests. +- `run_hook(module, payload, env)` — `subprocess.run([sys.executable, '-m', module], input=json.dumps(payload), ...)`, returns (rc, stdout, stderr). +- `real_home_canary` — autouse session tripwire (spec in §1F). +- Live-AWS: reuse `tests/integration/conftest.py` (`agentcore_throwaway_memories`, `bm_int_*` names, stale sweep, `BETTER_MEMORY_TEST_AGENTCORE_KEEP`). + +**pyproject additions** + +```toml +[tool.pytest.ini_options] +markers = [ + "integration: ...", # existing + "e2e: end-to-end clean-slate smoke tests (hermetic)",# new +] +# addopts unchanged: -m 'not integration' +``` + +--- + +## 3. Execution Model + +| invocation | runs | +|---|---| +| plain `pytest` | T1 + T2 (all `tests/e2e`, hermetic, offline) + `real_home_canary` tripwire + `tests/e2e_meta` fast checks (marker wiring, env-helper contract). setup.sh tests self-skip without bash; symlink asserts capability-probed; `e2e-setup-eof-aborts` self-skips on hosts with Ollama installed (runs on CI). | +| `pytest tests/e2e_meta/test_canary_home.py` / `test_env_bleed.py` | CI isolation-proof jobs (each re-runs the suite in a subprocess — too slow for every local run). | +| `BETTER_MEMORY_TEST_AGENTCORE=1 pytest -m integration tests/integration/test_agentcore_live_e2e.py` | T3 live AWS (~5–8 min, real cost; opt-in only). Without the env var: green-skipped. | +| `python scripts/e2e_mutation_smoke.py` | Manual/nightly suite-power validation (§5); never in default runs. | + +Offline guarantees on the default run: no DNS (`.invalid` / loopback literals only), no Ollama (zero-traffic tripwire), no AWS (endpoint + credential lockdown tripwire), no uv/network (setup.sh curl shim + primed/warm-cache preconditions). + +--- + +## 4. Product Bugs Discovered — Pinned as Current Behavior (FIXME markers) + +Each gets a test whose docstring names the defect and states the deletion/inversion condition. **These need product decisions/fixes; the tests will fail loudly when fixed:** + +1. **Vestigial ID-var gate** (`config.py:293-301`) — `BETTER_MEMORY_AGENTCORE_{SEMANTIC,EPISODIC}_MEMORY_ID` are required but their values are consumed by nothing (IDs come from `agentcore.json`); the error's own remediation (`agentcore init`) cannot clear it. Every documented agentcore setup dies pre-handshake. → `e2e-ac-neg-prehandshake-config-errors[idvar]` + delete-together dummy-var fixture. +2. **MCP dispatch gap in agentcore mode** (judge-discovered, empirically verified): `memory.observe`, `memory.semantic_observe`, `memory.record_use`, `memory.retrieve_observations` write/read **local sqlite**, never AWS; only `memory.retrieve` (and hooks) reach `AgentCoreBackend`. Agentcore users' memories silently land in a local file. → `e2e-ac-mcp-dispatch-gap-pin`. *This contradicts the phase-1 journey map and is the highest-priority product finding.* +3. **Hook-before-server ordering** — first SessionStart on a clean slate creates a schema-less `memory.db`, never writes the session marker, error-logging silently no-ops. → `e2e-sqlite-hook-before-server-degraded`. +4. **Region split-brain** — runtime factory signs with env `BETTER_MEMORY_AGENTCORE_REGION` (default eu-west-2); CLI/session_close use `agentcore.json`'s region. Silent cross-region divergence. → `e2e-ac-region-split-brain-pin`. +5. **Hook env-propagation gap** — installer writes no env into hook commands; `session_close` agentcore closure silently never fires unless the user exports the backend process-wide. → `e2e-ac-session-close-closure-and-env-gate` Case B. +6. **contextual_inject misconfig degradation** — `record_hook_error` creates a stray schema-less `memory.db` on clean slate; misconfigured agentcore users get a fully silent no-op on every prompt. → `e2e-ac-contextual-inject-wire-and-degradation` Cases B/C. +7. **boto3 missing = raw ImportError** — plain pip install + backend=agentcore gives no `better-memory[agentcore]` hint. → `e2e-ac-neg-boto3-missing` (inverted assertion). +8. **Corrupt `agentcore.json` error lacks remediation** (missing-file error has one; corrupt-file doesn't). → `e2e-ac-neg-corrupt-agentcore-json` negative pin. +9. **`knowledge-base/` never auto-created by the server** — pip installs get silently-empty knowledge search forever. → pinned in `e2e-sqlite-first-boot`. +10. **Orphan episode on failed observe** — Ollama-down observe commits a background episode then fails. → pinned in `e2e-sqlite-ollama-absent-default-backend`. +11. **setup.sh aborts headless at the Ollama prompt** (EOF under `set -euo pipefail`) before layout/install — partial install with zero diagnostics. → `e2e-setup-eof-aborts-all-or-nothing`. +12. **"No SQLite traffic" doc claim is false** — agentcore mode still creates+migrates both sqlite DBs. → pinned in `e2e-ac-server-boot-tools-hidden` and `e2e-ac-neg-missing-agentcore-json`. + +--- + +## 5. Mutation Smoke Checklist (validation phase) + +Driver `scripts/e2e_mutation_smoke.py`: runs inside a throwaway `git worktree add /mutsmoke HEAD` (never touches the live tree — required: repo currently has ~70 untracked scratch files); per mutation: apply patch → `pytest tests/e2e -q --tb=line -p no:cacheprovider` → assert red **with the expected node id in the failure output** and ≥1 other test still passing → revert via `git checkout --` on the target files; final unpatched control run must be green. + +| # | mutation (patch target) | must-flip sentinel | +|---|---|---| +| M1 | `mcp/server.py`: skip `apply_migrations` for memory.db at boot | `e2e-sqlite-first-boot-migrates-tools-knowledge` / observe journey (`no such table` via MCP tool error) | +| M2 | `hooks/session_bootstrap.py`: hoist `write_session_id()` above `service.bootstrap()` | `e2e-sqlite-hook-before-server-degraded` (`runtime/sessions` absent assertion) | +| M3 | `hooks/contextual_inject.py`: except path exits without printing the envelope | `e2e-sqlite-contextual-inject-contract` (exactly-one-JSON-line stdout) | +| M4 | drop hostile `tests/e2e/test_zz_seeded_breach.py` writing to `Path.home()/.claude.json` + `.better-memory/leak.txt` | `meta-canary-home-run` must FAIL with the isolation-breach/hash assertion naming `/.claude.json` (who-watches-the-watchers) — then green again after removal | + +CI adds a cheap "patches still apply cleanly" check without executing the suite. + +--- + +## 6. Ordered Implementation Task List + +Every task ≥90% confidence; where confidence rests on a mitigation, it is embedded. + +| # | task | deliverables | confidence | +|---|---|---|---| +| 1 | **Harness foundation**: `tests/e2e/_env.py` (`isolated_env` allowlist contract), `pyproject` `e2e` marker, `meta-env-helper-contract` tests (A/B/C/D), `meta-marker-tier-wiring`. | `_env.py`, `tests/e2e_meta/test_env_helper_contract.py`, `test_marker_wiring.py`, pyproject edit | 96% — pure-python fixtures mirroring the proven `mock_home` precedent; all assertions pre-verified by judges. | +| 2 | **Subprocess helpers**: `run_hook`, `mcp_session` (explicit USERPROFILE/HOME, errlog real-file, `cwd=` support), `clean_slate_home`. | `tests/e2e/conftest.py` (partial) | 95% — direct lift of `tests/mcp/test_server_integration.py` precedent + empirically verified SDK behaviors (McpError -32000, errlog fileno). | +| 3 | **Installer scenarios A1–A7**. | `test_install_hooks.py` | 95% — every assertion verified line-by-line against `install_hooks.py` by judges; symlink OSError via driver-script patch works on all hosts. | +| 4 | **sqlite journey C1–C6** (first-boot, degraded, heal, observe/record_use, session-close two-fire, spool-drain/synthesize) with all judge fixes (counts['created'], marker-key pinning via `cwd`, ratings payload shape, non-recursive glob). | `test_sqlite_journey.py`, `test_hooks_contracts.py` (partial) | 92% — long multi-step flows; mitigations embedded: exact wire shapes pinned in the design (no implementer guessing), each step's contract empirically verified in the verifier round. | +| 5 | **sqlite contracts C7–C8** (contextual_inject sqlite contract incl. default-mode; ollama-absent with <30s budget timed only around the call) + `e2e-ollama-zero-traffic-tripwire`. | `test_hooks_contracts.py`, `test_sqlite_negative.py`, `test_tripwires.py` (partial) | 93% — Case A/C empirically verified; Case B (injection fires) needs one seeding recipe — mitigation: seed via `memory.semantic_observe` through the server exactly as the session-close scenario does. | +| 6 | **Fake agentcore endpoint + env**: `_fake_agentcore.py` (gzip model routing), `_agentcore_env.py` (single dummy-ID location + FIXME), `e2e-aws-lockdown-tripwire`. | `_fake_agentcore.py`, `_agentcore_env.py`, `test_tripwires.py` | 91% — endpoint seam empirically proven (signed round-trip captured); residual risk is rest-json routing breadth — mitigation: route only the 8 operations the suite uses, with a fallback 200 `{}` default handler and per-test canned overrides. | +| 7 | **T2 agentcore D1–D7** (boot, retrieve fan-out, dispatch-gap pin, backend wire-fidelity, session_close, contextual_inject, region split-brain). | `test_agentcore_t2.py` | 90% — the dispatch-gap correction removes the false-premise scenarios; remaining wired paths (retrieve, hooks, direct backend) all empirically verified; mitigation for fan-out flake: order-insensitive set assertions, canned empty summaries. | +| 8 | **T2 negatives D8–D11** (parametrized pre-handshake, missing json + SDK level, corrupt json, boto3 shadow with content-based control). | `test_agentcore_neg.py` | 95% — every case empirically run during verification (3/3 stable); exact stderr substrings pinned to source. | +| 9 | **setup.sh B1–B4 + containment**: repo-copy execution, curl/ollama shims, env fixes (SYSTEMROOT/COMSPEC/TEMP/UV_CACHE_DIR), bash-version probe, real-`.venv`-unchanged assertion. | `test_setup_sh.py` | 90% — bash/uv host variability is the risk; mitigations embedded: skip-guards (no bash / host-ollama for the EOF test), shim-forced deterministic branches, warm-cache precondition, progress-marker assertions to localize failures. | +| 10 | **Harness meta**: `meta-canary-home-run` (case-insensitive outer strip, exit-5 vacuity), `real_home_canary` fixture (semantic subtree compare, worker-id canaries, import-time env snapshot), `meta-env-bleed-poisoned-shell`. | `tests/e2e_meta/test_canary_home.py`, `test_env_bleed.py`, conftest fixture | 92% — all judge fixes specified concretely; live-session false-alarm risk removed by the semantic-compare redesign. | +| 11 | **Existing-test additions** (from folded/killed scenarios): `tests/hooks/test_observer.py` — add `stdout == ''`, `no *.db under home`, full filename regex; `tests/cli/test_install_hooks.py::test_malformed_settings_leaves_claude_json_untouched` — add no-backup + no-`.tmp` assertions; strengthen `test_agentcore_unit` requestIdentifier to sha256 equality. | 3 small edits | 98% — one-line assertions into passing tests. | +| 12 | **T3 live E1–E2** (init/status journey; smoke + live retrieve + backend round-trip). | `tests/integration/test_agentcore_live_e2e.py` | 90% — reuses the proven throwaway-memories conftest; risks (cost, slowness, name collisions) mitigated by `bm_int_*` uuid names, pre-registered cleanup, stale sweep, opt-in gating; region set explicitly to dodge the split-brain. | +| 13 | **Mutation smoke** (`scripts/e2e_mutation_smoke.py` + M1–M4 patches, worktree-based) + CI patch-apply check. Run once as the validation gate for the whole phase. | script + 4 patches | 92% — worktree approach removes the dirty-tree blocker; patches anchored to verified line ranges; sentinel node ids fixed by tasks 4–5 and 10. | +| 14 | **Memory sweep + findings report**: record remaining product defects (§4) to better-memory (items 1–2 already recorded: obs `a912351f…`, `971703a0…`), file the dispatch-gap and idvar-gate as product issues. | observations + issue notes | 97%. | + +Sequencing constraint: tasks 1–2 block everything; 6 blocks 7–8; 13 runs last (its sentinels must exist). Tasks 3, 4–5, 9 are parallelizable after task 2. \ No newline at end of file From 1e28a2eeb0797a91cc277d99a68f4a670209a334 Mon Sep 17 00:00:00 2001 From: gethin Date: Sun, 12 Jul 2026 14:31:42 +0100 Subject: [PATCH 2/4] test: clean-slate e2e smoke suite (T1 sqlite, T2 agentcore-hermetic, T3 live-gated) Implements the phase-2 design (docs/superpowers/specs/2026-07-12-e2e-clean-slate-smoke-design.md): - tests/e2e/_env.py: isolated_env allowlist choke point (HOME+USERPROFILE+ BETTER_MEMORY_HOME redirect, case-insensitive strip of CLAUDE_*/ BETTER_MEMORY_*/AWS_*/OLLAMA_*) - tests/e2e/conftest.py: clean_slate_home, run_hook, mcp_session (explicit USERPROFILE/HOME for the mcp SDK Windows force-inherit), real_home_canary autouse tripwire, e2e marker auto-application - installer, setup.sh, sqlite journey, hook contracts, sqlite negatives, ollama zero-traffic tripwire scenarios - agentcore hermetic tier: _fake_agentcore.py (botocore-model-routed local endpoint via AWS_ENDPOINT_URL), _agentcore_env.py (single FIXME(idvar-gate) dummy-var location), T2 positives, negatives, AWS lockdown tripwire - tests/e2e_meta: env-helper contract (AST spawn-site provenance), marker/tier wiring, canary-home meta-run, env-bleed poisoned-shell - tests/integration/test_agentcore_live_e2e.py: T3 live journey (gated behind BETTER_MEMORY_TEST_AGENTCORE=1 + integration marker) - scripts/e2e_mutation_smoke.py + M1-M4: suite-power validation gate - 12 known product defects pinned as current behavior with FIXME docstrings Full suite: 1312 passed, 22 skipped, 21 deselected. Co-Authored-By: Claude Fable 5 --- pyproject.toml | 1 + scripts/e2e_mutation_smoke.py | 459 ++++++++++ tests/cli/test_install_hooks.py | 8 + tests/e2e/__init__.py | 1 + tests/e2e/_agentcore_env.py | 147 ++++ tests/e2e/_env.py | 135 +++ tests/e2e/_fake_agentcore.py | 272 ++++++ tests/e2e/conftest.py | 417 +++++++++ tests/e2e/test_agentcore_neg.py | 322 +++++++ tests/e2e/test_agentcore_t2.py | 850 +++++++++++++++++++ tests/e2e/test_hooks_contracts.py | 212 +++++ tests/e2e/test_install_hooks.py | 712 ++++++++++++++++ tests/e2e/test_setup_sh.py | 596 +++++++++++++ tests/e2e/test_sqlite_journey.py | 725 ++++++++++++++++ tests/e2e/test_sqlite_negative.py | 119 +++ tests/e2e/test_tripwires.py | 174 ++++ tests/e2e/test_tripwires_aws.py | 199 +++++ tests/e2e_meta/__init__.py | 1 + tests/e2e_meta/mutations/M1.patch | 13 + tests/e2e_meta/mutations/M2.patch | 35 + tests/e2e_meta/mutations/M3.patch | 14 + tests/e2e_meta/mutations/M4_seeded_breach.py | 67 ++ tests/e2e_meta/test_canary_home.py | 269 ++++++ tests/e2e_meta/test_env_bleed.py | 188 ++++ tests/e2e_meta/test_env_helper_contract.py | 492 +++++++++++ tests/e2e_meta/test_marker_wiring.py | 154 ++++ tests/hooks/test_observer.py | 11 + tests/integration/test_agentcore_live_e2e.py | 387 +++++++++ tests/storage/test_agentcore_unit.py | 7 +- 29 files changed, 6986 insertions(+), 1 deletion(-) create mode 100644 scripts/e2e_mutation_smoke.py create mode 100644 tests/e2e/__init__.py create mode 100644 tests/e2e/_agentcore_env.py create mode 100644 tests/e2e/_env.py create mode 100644 tests/e2e/_fake_agentcore.py create mode 100644 tests/e2e/conftest.py create mode 100644 tests/e2e/test_agentcore_neg.py create mode 100644 tests/e2e/test_agentcore_t2.py create mode 100644 tests/e2e/test_hooks_contracts.py create mode 100644 tests/e2e/test_install_hooks.py create mode 100644 tests/e2e/test_setup_sh.py create mode 100644 tests/e2e/test_sqlite_journey.py create mode 100644 tests/e2e/test_sqlite_negative.py create mode 100644 tests/e2e/test_tripwires.py create mode 100644 tests/e2e/test_tripwires_aws.py create mode 100644 tests/e2e_meta/__init__.py create mode 100644 tests/e2e_meta/mutations/M1.patch create mode 100644 tests/e2e_meta/mutations/M2.patch create mode 100644 tests/e2e_meta/mutations/M3.patch create mode 100644 tests/e2e_meta/mutations/M4_seeded_breach.py create mode 100644 tests/e2e_meta/test_canary_home.py create mode 100644 tests/e2e_meta/test_env_bleed.py create mode 100644 tests/e2e_meta/test_env_helper_contract.py create mode 100644 tests/e2e_meta/test_marker_wiring.py create mode 100644 tests/integration/test_agentcore_live_e2e.py diff --git a/pyproject.toml b/pyproject.toml index 595680b..9d9a379 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -51,6 +51,7 @@ asyncio_mode = "auto" addopts = "-m 'not integration'" markers = [ "integration: marks tests as integration tests (require external services)", + "e2e: end-to-end clean-slate smoke tests (hermetic)", ] [tool.ruff] diff --git a/scripts/e2e_mutation_smoke.py b/scripts/e2e_mutation_smoke.py new file mode 100644 index 0000000..978f70e --- /dev/null +++ b/scripts/e2e_mutation_smoke.py @@ -0,0 +1,459 @@ +"""Mutation smoke driver for the e2e clean-slate suite (design section 5, task 13). + +Proves ``tests/e2e`` (and its safety harness) still has teeth: each mutation +deliberately breaks one pinned contract inside a throwaway git worktree and +the suite must go RED with the expected sentinel node id in the failure +output while at least one other test still PASSES; after reverting, a final +unpatched control run must be GREEN. + +Everything runs in ``git worktree add --detach /wt HEAD`` — never in +the live tree (which carries scores of untracked scratch files and other +agents' uncommitted edits). Consequence: results reflect **HEAD**, not the +working tree; uncommitted changes are invisible to this gate. + +Mutations (artifacts live in ``tests/e2e_meta/mutations/``): + +* M1 ``M1.patch`` — ``better_memory/mcp/server.py``: skip ``apply_migrations`` + for memory.db at boot. Sentinel: ``test_first_boot_migrates_tools_knowledge``. +* M2 ``M2.patch`` — ``better_memory/hooks/session_bootstrap.py``: hoist + ``write_session_id`` above ``service.bootstrap()``. Sentinel: + ``test_hook_before_server_degraded`` (runtime/sessions-absent assertion). +* M3 ``M3.patch`` — ``better_memory/hooks/contextual_inject.py``: except path + exits without printing the envelope. Sentinel: + ``test_a_config_error_swallowed_envelope_still_printed`` (exactly-one-JSON-line). +* M4 ``M4_seeded_breach.py`` — NOT a patch: a hostile test file dropped in as + ``tests/e2e/test_zz_seeded_breach.py`` that writes into ``Path.home()`` + (armed via ``BM_E2E_SEEDED_BREACH=1``; refuses to run outside the seeded + canary home). The canary meta-run must FAIL its sentinel-hash / + file-set-diff assertions — who watches the watchers. + +Usage:: + + python scripts/e2e_mutation_smoke.py # full gate: M1-M4 + control + python scripts/e2e_mutation_smoke.py --mutation 2 # one mutation + control + python scripts/e2e_mutation_smoke.py --mutation 2 --no-control + python scripts/e2e_mutation_smoke.py --check-only # CI: patches apply cleanly + +Exit code 0 = every executed check passed; 1 = any failure (including a +mutation the suite did NOT catch — that is the whole point of this gate). +""" + +from __future__ import annotations + +import argparse +import os +import py_compile +import re +import shutil +import subprocess +import sys +import tempfile +from dataclasses import dataclass, field +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[1] +MUTATIONS_DIR = REPO_ROOT / "tests" / "e2e_meta" / "mutations" + +#: Per-pytest-run wall clock ceiling. The canary meta-run (M4) spawns an +#: inner pytest with its own 540s timeout; the full tests/e2e control run is +#: the longest leg. +PYTEST_TIMEOUT_SEC = 2400 +GIT_TIMEOUT_SEC = 120 + +#: Driver-owned control variables: never inherited from the outer shell, +#: only set per-mutation (M4 arms the breach through them). +_CONTROL_VARS = ("BM_E2E_SEEDED_BREACH", "BM_E2E_CANARY_INNER_SCOPE", "PYTEST_ADDOPTS") + +_FAILED_LINE_RE = re.compile(r"^(?:FAILED|ERROR)\s+\S+", re.MULTILINE) +_PASSED_RE = re.compile(r"(\d+) passed") + + +@dataclass(frozen=True) +class Mutation: + num: int + title: str + kind: str # "patch" | "dropin" + artifact: str # file name under tests/e2e_meta/mutations/ + #: repo-relative tracked files to revert with ``git checkout --`` (patch + #: kind) — the dropin kind reverts by deleting ``dropin_dest`` instead. + targets: tuple[str, ...] + pytest_args: tuple[str, ...] + #: The run is judged CAUGHT when ANY of these node-id substrings appears + #: in a FAILED/ERROR line of the pytest output. + expected_failed_nodes: tuple[str, ...] + dropin_dest: str = "" + extra_env: dict[str, str] = field(default_factory=dict) + + +MUTATIONS: tuple[Mutation, ...] = ( + Mutation( + num=1, + title="server.py: skip apply_migrations for memory.db at boot", + kind="patch", + artifact="M1.patch", + targets=("better_memory/mcp/server.py",), + pytest_args=("tests/e2e",), + expected_failed_nodes=("test_first_boot_migrates_tools_knowledge",), + ), + Mutation( + num=2, + title="session_bootstrap.py: write_session_id hoisted above bootstrap()", + kind="patch", + artifact="M2.patch", + targets=("better_memory/hooks/session_bootstrap.py",), + pytest_args=("tests/e2e",), + expected_failed_nodes=("test_hook_before_server_degraded",), + ), + Mutation( + num=3, + title="contextual_inject.py: except path exits without envelope", + kind="patch", + artifact="M3.patch", + targets=("better_memory/hooks/contextual_inject.py",), + pytest_args=("tests/e2e",), + expected_failed_nodes=( + "test_a_config_error_swallowed_envelope_still_printed", + ), + ), + Mutation( + num=4, + title="seeded breach test in tests/e2e — canary harness must catch it", + kind="dropin", + artifact="M4_seeded_breach.py", + targets=(), + # The sentinel here is the WATCHER, not tests/e2e: the canary + # meta-run reruns an inner slice (scoped below) under the seeded + # canary home and must flip red on its integrity assertions. + pytest_args=("tests/e2e_meta/test_canary_home.py",), + expected_failed_nodes=( + "test_sentinel_files_byte_identical", + "test_file_set_diff_is_empty", + ), + dropin_dest="tests/e2e/test_zz_seeded_breach.py", + extra_env={ + "BM_E2E_SEEDED_BREACH": "1", + # Reduced inner scope: one honest module (so >=1 inner test + # passes and the run is not vacuous) + the breach file. + "BM_E2E_CANARY_INNER_SCOPE": ( + "tests/e2e/test_hooks_contracts.py " + "tests/e2e/test_zz_seeded_breach.py" + ), + }, + ), +) + + +# --------------------------------------------------------------------------- infra + + +def _log(msg: str) -> None: + print(f"[mutation-smoke] {msg}", flush=True) + + +def _run( + cmd: list[str], + cwd: Path, + env: dict[str, str] | None = None, + timeout: int = GIT_TIMEOUT_SEC, +) -> subprocess.CompletedProcess[str]: + return subprocess.run( # noqa: S603 — dev tooling, fixed argv + cmd, + cwd=str(cwd), + env=env, + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + timeout=timeout, + ) + + +def _git(worktree_or_repo: Path, *args: str) -> subprocess.CompletedProcess[str]: + return _run(["git", "-C", str(worktree_or_repo), *args], cwd=REPO_ROOT) + + +def _pytest_env(mutation: Mutation | None) -> dict[str, str]: + env = dict(os.environ) + for key in list(env): + if key.upper() in _CONTROL_VARS: + del env[key] + if mutation is not None: + env.update(mutation.extra_env) + return env + + +def _tail(text: str, lines: int = 30) -> str: + return "\n".join(text.splitlines()[-lines:]) + + +class MutationSmokeError(Exception): + """A gate-level failure (setup problem or a mutation the suite missed).""" + + +# --------------------------------------------------------------------------- worktree + + +def _add_worktree(scratch: Path) -> Path: + worktree = scratch / "wt" + _log(f"git worktree add --detach {worktree} HEAD") + proc = _git(REPO_ROOT, "worktree", "add", "--detach", str(worktree), "HEAD") + if proc.returncode != 0: + raise MutationSmokeError(f"git worktree add failed:\n{proc.stderr}") + return worktree + + +def _remove_worktree(worktree: Path) -> None: + proc = _git(REPO_ROOT, "worktree", "remove", "--force", str(worktree)) + if proc.returncode != 0 and worktree.exists(): + # .venv locks etc. — brute-force the directory, then let git forget it. + shutil.rmtree(worktree, ignore_errors=True) + _git(REPO_ROOT, "worktree", "prune") + if worktree.exists(): + _log(f"WARNING: could not fully remove worktree at {worktree}") + + +def _assert_worktree_clean(worktree: Path, context: str) -> None: + """Tracked-file cleanliness gate (judge fix: --untracked-files=no).""" + proc = _git(worktree, "status", "--porcelain", "--untracked-files=no") + if proc.returncode != 0: + raise MutationSmokeError(f"git status failed in worktree:\n{proc.stderr}") + if proc.stdout.strip(): + raise MutationSmokeError( + f"worktree dirty {context}:\n{proc.stdout}" + ) + + +def _require_sentinel_suite(worktree: Path) -> None: + missing = [ + rel + for rel in ("tests/e2e", "tests/e2e_meta/test_canary_home.py") + if not (worktree / rel).exists() + ] + if missing: + raise MutationSmokeError( + f"missing at HEAD: {missing} — the mutation gate runs against a " + "worktree of HEAD, so the e2e suite must be committed first" + ) + + +# --------------------------------------------------------------------------- apply / revert + + +def _apply(worktree: Path, mutation: Mutation) -> None: + artifact = MUTATIONS_DIR / mutation.artifact + if not artifact.is_file(): + raise MutationSmokeError(f"mutation artifact missing: {artifact}") + if mutation.kind == "patch": + proc = _git(worktree, "apply", "--whitespace=nowarn", str(artifact)) + if proc.returncode != 0: + raise MutationSmokeError( + f"M{mutation.num} failed to apply:\n{proc.stderr}" + ) + else: # dropin + dest = worktree / mutation.dropin_dest + dest.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(artifact, dest) + + +def _revert(worktree: Path, mutation: Mutation) -> None: + if mutation.kind == "patch": + proc = _git(worktree, "checkout", "--", *mutation.targets) + if proc.returncode != 0: + raise MutationSmokeError( + f"M{mutation.num} revert failed:\n{proc.stderr}" + ) + else: + dest = worktree / mutation.dropin_dest + if dest.exists(): + dest.unlink() + _assert_worktree_clean(worktree, f"after reverting M{mutation.num}") + + +# --------------------------------------------------------------------------- pytest + + +def _pytest( + worktree: Path, args: tuple[str, ...], mutation: Mutation | None +) -> subprocess.CompletedProcess[str]: + # ``uv run`` inside the worktree provisions the worktree's own .venv + # (editable install of the PATCHED sources) — never the live repo's venv, + # whose editable install points at the unpatched tree. + cmd = ["uv", "run", "pytest", *args, "-q", "--tb=line", "-p", "no:cacheprovider"] + _log(" " + " ".join(cmd)) + return _run( + cmd, + cwd=worktree, + env=_pytest_env(mutation), + timeout=PYTEST_TIMEOUT_SEC, + ) + + +def _judge_red_run( + mutation: Mutation, proc: subprocess.CompletedProcess[str] +) -> list[str]: + """Return a list of problems (empty = the suite caught the mutation).""" + out = proc.stdout + "\n" + proc.stderr + problems: list[str] = [] + if proc.returncode == 0: + problems.append("suite stayed GREEN — mutation not caught") + if proc.returncode == 5: + problems.append("exit 5: nothing collected — sentinel run is vacuous") + failed_lines = _FAILED_LINE_RE.findall(out) + if not any( + node in line for line in failed_lines for node in mutation.expected_failed_nodes + ): + problems.append( + "expected sentinel node id not in failure output " + f"(any of {list(mutation.expected_failed_nodes)}); " + f"FAILED lines seen: {failed_lines or 'none'}" + ) + passed = sum(int(n) for n in _PASSED_RE.findall(out)) + if passed < 1: + problems.append( + f"no other test passed ({passed} passed) — cannot rule out a " + "harness-wide breakage masquerading as a caught mutation" + ) + return problems + + +# --------------------------------------------------------------------------- modes + + +def run_check_only(worktree: Path) -> int: + """CI cheap check: every patch applies cleanly; the dropin compiles.""" + failures = 0 + for mutation in MUTATIONS: + artifact = MUTATIONS_DIR / mutation.artifact + if not artifact.is_file(): + _log(f"M{mutation.num}: FAIL — artifact missing: {artifact}") + failures += 1 + continue + if mutation.kind == "patch": + proc = _git(worktree, "apply", "--check", "--whitespace=nowarn", str(artifact)) + if proc.returncode != 0: + _log(f"M{mutation.num}: FAIL — patch does not apply:\n{proc.stderr}") + failures += 1 + else: + _log(f"M{mutation.num}: OK (applies cleanly)") + else: + # cfile under the throwaway worktree: never litter the live + # repo's tests/e2e_meta/mutations/__pycache__. + cfile = worktree / f".m{mutation.num}-dropin.pyc" + try: + py_compile.compile(str(artifact), cfile=str(cfile), doraise=True) + except py_compile.PyCompileError as exc: + _log(f"M{mutation.num}: FAIL — dropin does not compile: {exc}") + failures += 1 + else: + _log(f"M{mutation.num}: OK (dropin compiles)") + return failures + + +def run_mutation(worktree: Path, mutation: Mutation) -> list[str]: + _log(f"M{mutation.num}: {mutation.title}") + _apply(worktree, mutation) + try: + proc = _pytest(worktree, mutation.pytest_args, mutation) + problems = _judge_red_run(mutation, proc) + if problems: + _log(f"M{mutation.num}: NOT CAUGHT") + for problem in problems: + _log(f" - {problem}") + _log(" --- pytest tail ---") + for line in _tail(proc.stdout).splitlines(): + _log(f" | {line}") + else: + _log(f"M{mutation.num}: caught (rc={proc.returncode})") + return problems + finally: + _revert(worktree, mutation) + + +def run_control(worktree: Path, include_canary: bool) -> list[str]: + args: tuple[str, ...] = ("tests/e2e",) + if include_canary: + # M4's "then green again after removal" leg (design section 5). + args = ("tests/e2e", "tests/e2e_meta/test_canary_home.py") + _log("control run (unpatched)") + proc = _pytest(worktree, args, mutation=None) + if proc.returncode != 0: + return [ + f"control run rc={proc.returncode} — unpatched suite is not green:\n" + f"{_tail(proc.stdout)}\n{_tail(proc.stderr, 10)}" + ] + _log("control run: green") + return [] + + +# --------------------------------------------------------------------------- main + + +def _parse_args(argv: list[str]) -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument( + "--mutation", + type=int, + choices=sorted(m.num for m in MUTATIONS), + help="run a single mutation instead of all four", + ) + parser.add_argument( + "--check-only", + action="store_true", + help="only verify the patches apply cleanly at HEAD (no test execution)", + ) + parser.add_argument( + "--no-control", + action="store_true", + help="skip the final unpatched control run (iteration aid)", + ) + return parser.parse_args(argv) + + +def main(argv: list[str] | None = None) -> int: + args = _parse_args(sys.argv[1:] if argv is None else argv) + + dirty = _git(REPO_ROOT, "status", "--porcelain", "--untracked-files=no") + if dirty.stdout.strip(): + _log("NOTE: tracked files are modified in the live tree; this gate") + _log(" runs against HEAD — those modifications are NOT exercised:") + for line in dirty.stdout.strip().splitlines(): + _log(f" {line}") + + scratch = Path(tempfile.mkdtemp(prefix="bm-mutsmoke-")) + worktree: Path | None = None + failures: list[str] = [] + try: + worktree = _add_worktree(scratch) + + if args.check_only: + return 1 if run_check_only(worktree) else 0 + + _require_sentinel_suite(worktree) + selected = [m for m in MUTATIONS if args.mutation in (None, m.num)] + for mutation in selected: + failures.extend( + f"M{mutation.num}: {problem}" + for problem in run_mutation(worktree, mutation) + ) + if not args.no_control: + failures.extend( + run_control(worktree, include_canary=any(m.num == 4 for m in selected)) + ) + except MutationSmokeError as exc: + failures.append(str(exc)) + finally: + if worktree is not None: + _remove_worktree(worktree) + shutil.rmtree(scratch, ignore_errors=True) + + if failures: + _log(f"RESULT: FAIL ({len(failures)} problem(s))") + for failure in failures: + _log(f" * {failure}") + return 1 + _log("RESULT: OK — every executed mutation was caught; control green") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/cli/test_install_hooks.py b/tests/cli/test_install_hooks.py index c5449e4..336c9ab 100644 --- a/tests/cli/test_install_hooks.py +++ b/tests/cli/test_install_hooks.py @@ -558,6 +558,14 @@ def test_malformed_settings_leaves_claude_json_untouched( assert settings.read_text(encoding="utf-8") == '{"hooks": {invalid' # Critical: claude.json was NOT written. assert not claude_json.exists() + # A refused install takes no backup: neither _backup nor + # _atomic_write ever started. (The fixture pre-creates the empty + # install-backups dir, so pin its contents, not its existence.) + backups = mock_home / ".better-memory" / "install-backups" + assert not any(backups.iterdir()) + # ... and leaves no .tmp litter next to (or anywhere near) either + # target — _atomic_write writes {target}.tmp before os.replace. + assert not list(mock_home.rglob("*.tmp")) def test_summary_lines_printed_on_success(self, mock_home: Path) -> None: result = _run_cli(mock_home) diff --git a/tests/e2e/__init__.py b/tests/e2e/__init__.py new file mode 100644 index 0000000..5823c4f --- /dev/null +++ b/tests/e2e/__init__.py @@ -0,0 +1 @@ +"""Hermetic end-to-end clean-slate smoke tests (T1 sqlite + T2 fake-agentcore).""" diff --git a/tests/e2e/_agentcore_env.py b/tests/e2e/_agentcore_env.py new file mode 100644 index 0000000..9ef4656 --- /dev/null +++ b/tests/e2e/_agentcore_env.py @@ -0,0 +1,147 @@ +"""agentcore-mode extension of the e2e env choke point. + +``agentcore_env`` layers the T2 lockdown contract on top of +``tests.e2e._env.isolated_env`` (everything still flows through the single +allowlist choke point, so the e2e_meta contract tests bless spawns whose +env came from here): + +* ``BETTER_MEMORY_STORAGE_BACKEND=agentcore`` + a pinned test region; +* ``AWS_ENDPOINT_URL`` → the local :class:`tests.e2e._fake_agentcore. + FakeAgentCore` port, so every boto3 request from every child process + lands on loopback (verified for both agentcore planes at boto3 1.43.14); +* credential lockdown: fake static keys (``bm-e2e-fake``), shared + credentials / config files pointed at *nonexistent* tmp paths, IMDS + disabled — the SigV4 ``Credential=bm-e2e-fake/...`` scope on captured + requests proves the default chain never reached real creds; +* ``AWS_IGNORE_CONFIGURED_ENDPOINT_URLS`` (and every other outer ``AWS_*`` + var) never survives: ``isolated_env`` builds from an allowlist and drops + all ``AWS_*`` keys case-insensitively. + +This module is THE ONLY place the two dummy memory-ID vars are set — see +the FIXME below. ``tests/e2e/test_tripwires_aws.py`` grep-pins that +exclusivity. +""" + +from __future__ import annotations + +from pathlib import Path + +from better_memory.storage.agentcore_persistence import ( + AgentCoreConfig, + MemoryRecord, + save_agentcore_config, +) +from tests.e2e._env import isolated_env + +#: Fake static credentials. The value is asserted inside SigV4 Credential +#: scopes by the AWS lockdown tripwire — keep them greppable and unique. +FAKE_ACCESS_KEY_ID = "bm-e2e-fake" +FAKE_SECRET_ACCESS_KEY = "bm-e2e-fake-secret" + +#: Region pinned into the child env (BETTER_MEMORY_AGENTCORE_REGION). Note +#: this matches the product default in better_memory/config.py so removing +#: the pin (pin=None) still signs eu-west-2 — the region-split-brain test +#: relies on that. +DEFAULT_TEST_REGION = "eu-west-2" + +# FIXME(idvar-gate): the two BETTER_MEMORY_AGENTCORE_*_MEMORY_ID env vars +# below are a WORKAROUND for the dead presence-only gate at +# better_memory/config.py:293-301. Their values are consumed by NOTHING — +# runtime memory IDs come exclusively from agentcore.json (see +# better_memory/storage/factory.py; the region-split-brain test proves the +# dummies never reach the wire). This is the ONLY location that sets them +# (grep-pinned by tests/e2e/test_tripwires_aws.py). DELETE these two +# entries together with tests/e2e/test_agentcore_neg.py's idvar-gate +# defect-pin test when the product fix (remove the gate, or repoint it at +# agentcore.json existence) lands. +DUMMY_ID_VARS: dict[str, str] = { + "BETTER_MEMORY_AGENTCORE_SEMANTIC_MEMORY_ID": "DUMMY-SEM", + "BETTER_MEMORY_AGENTCORE_EPISODIC_MEMORY_ID": "DUMMY-EPI", +} + +#: Dummy values, exported so wire-absence assertions ("the dead env values +#: never appear in any request") don't have to hardcode them. +DUMMY_SEMANTIC_MEMORY_ID = DUMMY_ID_VARS["BETTER_MEMORY_AGENTCORE_SEMANTIC_MEMORY_ID"] +DUMMY_EPISODIC_MEMORY_ID = DUMMY_ID_VARS["BETTER_MEMORY_AGENTCORE_EPISODIC_MEMORY_ID"] + + +def remove_dummy_id_vars_pins() -> dict[str, None]: + """Pins that REMOVE the two dummy ID vars (misconfig scenarios). + + Kwarg-splat into :func:`agentcore_env` so tests never spell the var + names out (the tripwire grep-pins the literals to this module plus the + one defect-pin test):: + + env = agentcore_env(home, fake.port, **remove_dummy_id_vars_pins()) + """ + return dict.fromkeys(DUMMY_ID_VARS) + + +def agentcore_env( + tmp_home: Path, fake_port: int, **pins: str | None +) -> dict[str, str]: + """Hermetic agentcore-mode child env homed at ``tmp_home``. + + ``fake_port`` is the local :class:`FakeAgentCore` port. ``pins`` + override or extend the defaults exactly like ``isolated_env`` pins + (``None`` removes a key — e.g. ``BETTER_MEMORY_STORAGE_BACKEND=None`` + for the session-close env-gate case).""" + defaults: dict[str, str | None] = { + "BETTER_MEMORY_STORAGE_BACKEND": "agentcore", + "BETTER_MEMORY_AGENTCORE_REGION": DEFAULT_TEST_REGION, + "AWS_ENDPOINT_URL": f"http://127.0.0.1:{fake_port}", + "AWS_ACCESS_KEY_ID": FAKE_ACCESS_KEY_ID, + "AWS_SECRET_ACCESS_KEY": FAKE_SECRET_ACCESS_KEY, + # Nonexistent-by-construction paths (never created by anything): + # the default credential/config chain finds nothing outside tmp. + "AWS_SHARED_CREDENTIALS_FILE": str( + Path(tmp_home) / "no-such-aws-credentials" + ), + "AWS_CONFIG_FILE": str(Path(tmp_home) / "no-such-aws-config"), + "AWS_EC2_METADATA_DISABLED": "true", + **DUMMY_ID_VARS, # FIXME(idvar-gate) — see module comment above. + } + defaults.update(pins) + return isolated_env(tmp_home, **defaults) + + +def write_fake_agentcore_json( + bm_home: Path, + *, + region: str = DEFAULT_TEST_REGION, + episodic_memory_id: str = "EPI-FAKE-0001", + semantic_memory_id: str = "SEM-FAKE-0001", + episodic_strategy_id: str = "STRAT-EPI-1", + semantic_strategy_id: str = "STRAT-SEM-1", +) -> AgentCoreConfig: + """Fabricate a schema-1 ``agentcore.json`` under ``bm_home`` (the + BETTER_MEMORY_HOME dir, i.e. ``/.better-memory``) via the real + ``save_agentcore_config`` writer, and return the config object.""" + cfg = AgentCoreConfig( + schema_version=1, + region=region, + semantic=MemoryRecord( + memory_id=semantic_memory_id, + memory_arn=( + f"arn:aws:bedrock-agentcore:{region}:000000000000:" + f"memory/{semantic_memory_id}" + ), + memory_name="bm_e2e_semantic", + strategy_id=semantic_strategy_id, + strategy_name="bm_e2e_semantic_strategy", + event_expiry_duration_days=365, + ), + episodic=MemoryRecord( + memory_id=episodic_memory_id, + memory_arn=( + f"arn:aws:bedrock-agentcore:{region}:000000000000:" + f"memory/{episodic_memory_id}" + ), + memory_name="bm_e2e_episodic", + strategy_id=episodic_strategy_id, + strategy_name="bm_e2e_episodic_strategy", + event_expiry_duration_days=90, + ), + ) + save_agentcore_config(cfg, bm_home) + return cfg diff --git a/tests/e2e/_env.py b/tests/e2e/_env.py new file mode 100644 index 0000000..cabfe4c --- /dev/null +++ b/tests/e2e/_env.py @@ -0,0 +1,135 @@ +"""THE single environment choke point for every e2e subprocess spawn. + +Every subprocess / stdio-server spawn in ``tests/e2e`` MUST build its +environment through :func:`isolated_env` (or ``_agentcore_env.agentcore_env``, +which extends it). Hand-rolled env dicts and ``{**os.environ, ...}`` patterns +are banned — enforced by ``tests/e2e_meta/test_env_helper_contract.py``. + +Contract (design section 2, ``2026-07-12-e2e-clean-slate-smoke-design.md``): + +* Built from an **allowlist**, never ``os.environ.copy()``. Only + ``{PATH, SYSTEMROOT, COMSPEC, PATHEXT, TEMP, TMP, WINDIR, LANG, LC_ALL, + PYTHONIOENCODING}`` are preserved from the outer environment, matched + **case-insensitively** (Windows env keys carry arbitrary case: + ``SystemRoot``, ``Path``, ``windir``...). +* ``HOME`` and ``USERPROFILE`` are BOTH set, unconditionally, on every OS. + A POSIX contributor deleting the "redundant" USERPROFILE line is the exact + bug that torches a Windows dev's real ``~/.claude.json`` — do not touch. +* ``HOMEDRIVE``/``HOMEPATH`` are pinned to the tmp home on Windows + (``Path.home()``'s fallback chain). +* All outer ``CLAUDE_*`` / ``BETTER_MEMORY_*`` / ``AWS_*`` / ``OLLAMA_*`` + vars are dropped case-insensitively (automatic: allowlist construction + never copies them in the first place). +* Pins: ``BETTER_MEMORY_HOME=/.better-memory``, ``BETTER_MEMORY_PROJECT``, + ``CLAUDE_SESSION_ID``, ``BETTER_MEMORY_EMBEDDINGS_BACKEND=sqlite`` and a + poisoned ``OLLAMA_HOST`` so nothing can ever reach a real Ollama daemon. +""" + +from __future__ import annotations + +import os +import sys +from pathlib import Path + +#: Outer-environment variables that survive into the isolated env. +#: SYSTEMROOT/COMSPEC: child pythons on Windows fail to import ssl/sqlite3 +#: without them. TEMP/TMP: tempfile.gettempdir() must not fall back to CWD. +ALLOWLIST: frozenset[str] = frozenset( + { + "PATH", + "SYSTEMROOT", + "COMSPEC", + "PATHEXT", + "TEMP", + "TMP", + "WINDIR", + "LANG", + "LC_ALL", + "PYTHONIOENCODING", + } +) + +#: Prefixes that must never leak from the outer environment (checked +#: case-insensitively). The allowlist construction guarantees this; the +#: constant is exported so meta tests can assert against the same source +#: of truth. +STRIPPED_PREFIXES: tuple[str, ...] = ( + "CLAUDE_", + "BETTER_MEMORY_", + "AWS_", + "OLLAMA_", +) + +#: Poisoned Ollama endpoint: `.invalid` is an IETF-reserved TLD, so no DNS +#: resolution can ever succeed — any code path that tries to reach Ollama +#: fails fast instead of touching a real daemon. +POISONED_OLLAMA_HOST = "http://does-not-exist.invalid:1" + +DEFAULT_PROJECT = "e2e-project" +DEFAULT_SESSION_ID = "e2e-session-1" + + +def _set(env: dict[str, str], key: str, value: str) -> None: + """Set ``key`` replacing any case-insensitive duplicate first.""" + upper = key.upper() + for existing in [k for k in env if k.upper() == upper]: + del env[existing] + env[key] = value + + +def _delete(env: dict[str, str], key: str) -> None: + upper = key.upper() + for existing in [k for k in env if k.upper() == upper]: + del env[existing] + + +def isolated_env(tmp_home: Path, **pins: str | None) -> dict[str, str]: + """Build a hermetic child-process environment homed at ``tmp_home``. + + ``pins`` override or extend the defaults; a pin of ``None`` removes the + key entirely (e.g. ``CLAUDE_SESSION_ID=None`` for marker-bridge tests). + + Returns a plain dict suitable for ``subprocess.run(env=...)`` and + ``StdioServerParameters(env=...)``. + """ + home = str(tmp_home) + env: dict[str, str] = {} + + # 1. Allowlist copy from the outer env, case-insensitive, deduped. + seen: set[str] = set() + for key, value in os.environ.items(): + upper = key.upper() + if upper in ALLOWLIST and upper not in seen: + seen.add(upper) + env[key] = value + + # 2. Home redirection — both vars, every OS (see module docstring). + _set(env, "HOME", home) + _set(env, "USERPROFILE", home) + if sys.platform == "win32": + drive, tail = os.path.splitdrive(home) + if drive: + _set(env, "HOMEDRIVE", drive) + _set(env, "HOMEPATH", tail or "\\") + + # 3. Deliberate better-memory pins. + _set(env, "BETTER_MEMORY_HOME", str(Path(tmp_home) / ".better-memory")) + _set(env, "BETTER_MEMORY_PROJECT", DEFAULT_PROJECT) + _set(env, "CLAUDE_SESSION_ID", DEFAULT_SESSION_ID) + _set(env, "BETTER_MEMORY_EMBEDDINGS_BACKEND", "sqlite") + _set(env, "OLLAMA_HOST", POISONED_OLLAMA_HOST) + + # 4. Caller pins (override defaults; None removes). + for key, value in pins.items(): + if value is None: + _delete(env, key) + else: + _set(env, key, value) + + # 5. Invariant: case-insensitive key uniqueness (duplicate 'Path'/'PATH' + # keys corrupt Windows CreateProcess environment blocks). + uppers = [k.upper() for k in env] + if len(set(uppers)) != len(uppers): + raise AssertionError(f"case-insensitive duplicate keys in isolated_env: {sorted(env)}") + + return env diff --git a/tests/e2e/_fake_agentcore.py b/tests/e2e/_fake_agentcore.py new file mode 100644 index 0000000..1791a09 --- /dev/null +++ b/tests/e2e/_fake_agentcore.py @@ -0,0 +1,272 @@ +"""Local fake AWS Bedrock AgentCore endpoint for the hermetic T2 suite. + +A ``ThreadingHTTPServer`` bound to ``127.0.0.1`` on an ephemeral port. The +operation routing table is derived from the *installed* botocore service +models (``botocore/data/bedrock-agentcore*/**/service-2.json.gz``, gzip +decoded — the models ship gzipped at boto3/botocore 1.43.x), so path +matching is pinned to the real wire contract, never guessed. + +Behavior: + +* every request is recorded as a :class:`RecordedRequest` + ``(operation, method, path, headers, body)`` — synchronously, before the + response is written, so tests can assert immediately after the client + call returns; +* responses are canned per operation name via :meth:`FakeAgentCore. + set_response` (a dict, or a callable ``RecordedRequest -> dict`` for + per-request behavior); +* any un-canned (or unroutable) request gets a ``200 {}`` fallback — + botocore's rest-json parser tolerates missing output members, and the + repo's backend code uses ``response.get(...)`` throughout. + +Plain HTTP, no TLS — verified end-to-end against boto3 1.43.14 (the +``AWS_ENDPOINT_URL`` seam covers both the ``bedrock-agentcore`` data plane +and the ``bedrock-agentcore-control`` control plane). + +Usage:: + + with FakeAgentCore() as fake: + fake.set_response("ListMemoryRecords", {"memoryRecordSummaries": []}) + env = agentcore_env(clean_slate_home, fake.port) + ... + assert len(fake.requests_for("ListMemoryRecords")) == 3 +""" + +from __future__ import annotations + +import gzip +import json +import re +import threading +from dataclasses import dataclass, field +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path +from typing import Any + +import botocore + +#: Both planes' models, relative to the botocore package dir. The trailing +#: ``.gz`` is load-bearing: the installed models are gzipped (judge fix — +#: a plain ``service-2.json`` glob finds nothing at botocore 1.43.14). +_MODEL_GLOB = "data/bedrock-agentcore*/*/service-2.json.gz" + +_PLACEHOLDER_RE = re.compile(r"\{([^}]+)\}") + + +def _uri_to_regex(request_uri: str) -> re.Pattern[str]: + """Compile a botocore ``requestUri`` (e.g. ``/memories/{memoryId}/events``) + into an anchored path regex. ``{param}`` matches one path segment; + ``{param+}`` (greedy) matches across segments.""" + path = request_uri.split("?", 1)[0] + parts: list[str] = [] + pos = 0 + for match in _PLACEHOLDER_RE.finditer(path): + parts.append(re.escape(path[pos : match.start()])) + parts.append(".+" if match.group(1).endswith("+") else "[^/]+") + pos = match.end() + parts.append(re.escape(path[pos:])) + return re.compile("^" + "".join(parts) + "$") + + +def _load_routes() -> list[tuple[str, str, re.Pattern[str]]]: + """Build ``(operation_name, http_method, path_regex)`` routes from the + installed gzipped botocore models for both agentcore planes.""" + data_root = Path(botocore.__file__).resolve().parent + model_paths = sorted(data_root.glob(_MODEL_GLOB)) + if not model_paths: + raise FileNotFoundError( + f"no bedrock-agentcore service models found under {data_root} " + f"(glob {_MODEL_GLOB!r}) — is the [agentcore] extra installed?" + ) + routes: list[tuple[str, str, re.Pattern[str]]] = [] + for model_path in model_paths: + model = json.loads(gzip.decompress(model_path.read_bytes())) + for op_name, op in model.get("operations", {}).items(): + http = op.get("http", {}) + request_uri = http.get("requestUri") + if not request_uri: + continue + routes.append( + (op_name, http.get("method", "POST"), _uri_to_regex(request_uri)) + ) + return routes + + +_SIGV4_CREDENTIAL_RE = re.compile(r"Credential=([^,\s]+)") + + +@dataclass +class RecordedRequest: + """One captured HTTP request, matched to a botocore operation name.""" + + operation: str | None + method: str + path: str # includes any query string + headers: dict[str, str] = field(default_factory=dict) + body: Any = None # parsed JSON when possible, raw text otherwise, None if empty + + def header(self, name: str) -> str: + """Case-insensitive header lookup ('' when absent).""" + lower = name.lower() + for key, value in self.headers.items(): + if key.lower() == lower: + return value + return "" + + @property + def host(self) -> str: + return self.header("Host") + + @property + def authorization(self) -> str: + return self.header("Authorization") + + @property + def sigv4_access_key(self) -> str: + """Access-key id from the SigV4 ``Credential=`` scope ('' if unsigned).""" + scope = self._credential_scope() + return scope[0] if scope else "" + + @property + def sigv4_region(self) -> str: + """Signing region from the SigV4 ``Credential=`` scope ('' if unsigned). + + Scope shape: ``////aws4_request`` + — the region is positional (index 2).""" + scope = self._credential_scope() + return scope[2] if scope and len(scope) >= 3 else "" + + def _credential_scope(self) -> list[str] | None: + match = _SIGV4_CREDENTIAL_RE.search(self.authorization) + return match.group(1).split("/") if match else None + + def text(self) -> str: + """path + serialized body, for model-independent substring asserts + ('is this memory id anywhere on the wire?').""" + if self.body is None: + body_text = "" + elif isinstance(self.body, str): + body_text = self.body + else: + body_text = json.dumps(self.body) + return f"{self.path} {body_text}" + + +class FakeAgentCore: + """Recording fake for both bedrock-agentcore planes. Context manager.""" + + def __init__(self) -> None: + self._routes = _load_routes() + self._known_ops = {name for name, _, _ in self._routes} + self._responses: dict[str, Any] = {} + self._lock = threading.Lock() + self.requests: list[RecordedRequest] = [] + + fake = self + + class _Handler(BaseHTTPRequestHandler): + # HTTP/1.1 + explicit Content-Length so urllib3's keep-alive + # connection reuse works against this server. + protocol_version = "HTTP/1.1" + + def _handle(self) -> None: + fake._serve(self) + + do_GET = _handle + do_POST = _handle + do_PUT = _handle + do_DELETE = _handle + do_PATCH = _handle + + def log_message(self, *args: Any) -> None: # silence stderr + pass + + self._server = ThreadingHTTPServer(("127.0.0.1", 0), _Handler) + self._thread = threading.Thread( + target=self._server.serve_forever, name="fake-agentcore", daemon=True + ) + self._thread.start() + + # ----- test-facing API ----- + + @property + def port(self) -> int: + return int(self._server.server_address[1]) + + @property + def endpoint_url(self) -> str: + return f"http://127.0.0.1:{self.port}" + + def set_response(self, operation: str, payload: Any) -> None: + """Can a response for an operation name (e.g. ``"CreateEvent"``). + + ``payload`` is a JSON-serializable dict, or a callable + ``RecordedRequest -> dict``. Raises on unknown operation names so a + typo cannot silently leave the fallback ``{}`` in place.""" + if operation not in self._known_ops: + raise ValueError( + f"unknown agentcore operation {operation!r}; " + f"known: {sorted(self._known_ops)}" + ) + self._responses[operation] = payload + + def requests_for(self, operation: str) -> list[RecordedRequest]: + with self._lock: + return [r for r in self.requests if r.operation == operation] + + def clear(self) -> None: + """Drop all recorded requests (canned responses are kept).""" + with self._lock: + self.requests.clear() + + def close(self) -> None: + self._server.shutdown() + self._server.server_close() + self._thread.join(timeout=10) + + def __enter__(self) -> FakeAgentCore: + return self + + def __exit__(self, *exc_info: object) -> None: + self.close() + + # ----- server side ----- + + def _serve(self, handler: BaseHTTPRequestHandler) -> None: + length = int(handler.headers.get("Content-Length") or 0) + raw = handler.rfile.read(length) if length else b"" + + path_only = handler.path.split("?", 1)[0] + operation: str | None = None + for name, method, pattern in self._routes: + if method == handler.command and pattern.match(path_only): + operation = name + break + + body: Any = None + if raw: + try: + body = json.loads(raw) + except (ValueError, UnicodeDecodeError): + body = raw.decode("utf-8", errors="replace") + + record = RecordedRequest( + operation=operation, + method=handler.command, + path=handler.path, + headers=dict(handler.headers.items()), + body=body, + ) + with self._lock: + self.requests.append(record) + + payload = self._responses.get(operation or "", {}) + if callable(payload): + payload = payload(record) + data = json.dumps(payload).encode("utf-8") + + handler.send_response(200) + handler.send_header("Content-Type", "application/json") + handler.send_header("Content-Length", str(len(data))) + handler.end_headers() + handler.wfile.write(data) diff --git a/tests/e2e/conftest.py b/tests/e2e/conftest.py new file mode 100644 index 0000000..fad19d7 --- /dev/null +++ b/tests/e2e/conftest.py @@ -0,0 +1,417 @@ +"""Shared fixtures and subprocess helpers for the hermetic e2e suite. + +Provides: + +* ``clean_slate_home`` — a brand-new fake user home (empty directory). +* ``run_hook`` — spawn a better_memory hook module exactly as Claude Code + would (stdin JSON payload, captured stdout/stderr). +* ``mcp_session`` — async context manager driving the real MCP stdio server + via the ``mcp`` client SDK with hermetic env handling. +* auto-marking: every test collected under ``tests/e2e`` gets the ``e2e`` + marker (declared in pyproject) so module authors cannot forget it. + +``real_home_canary`` (the autouse session tripwire from design section 1F) +lives at the bottom of this file: it plants additive dot-canaries in the +REAL home and semantically compares the better-memory config subtrees at +session teardown, turning a single hand-rolled-env spawn into an immediate +local failure instead of corrupted personal Claude config days later. + +Import helpers from test modules as:: + + from tests.e2e.conftest import mcp_session, run_hook +""" + +from __future__ import annotations + +import hashlib +import json +import os +import subprocess +import sys +import tempfile +import uuid +import warnings +from collections.abc import AsyncIterator, Iterator +from contextlib import asynccontextmanager +from datetime import timedelta +from pathlib import Path +from typing import IO, Any + +import pytest +from mcp import ClientSession +from mcp.client.stdio import StdioServerParameters, stdio_client + +_E2E_DIR = Path(__file__).resolve().parent + +#: Default MCP server invocation (the real production entry point). +SERVER_ARGS: tuple[str, ...] = ("-m", "better_memory.mcp") + +#: Generous read timeout: first boot runs migrations + knowledge reindex. +DEFAULT_READ_TIMEOUT = timedelta(seconds=60) + + +def pytest_collection_modifyitems(items: list[pytest.Item]) -> None: + """Auto-apply the ``e2e`` marker to everything under tests/e2e.""" + for item in items: + try: + path = Path(str(item.path)) + except Exception: + continue + if _E2E_DIR in path.parents or path.parent == _E2E_DIR: + item.add_marker(pytest.mark.e2e) + + +@pytest.fixture +def clean_slate_home(tmp_path: Path) -> Path: + """A truly clean-slate fake user home. + + The directory itself exists (a real user's home always does) but is + completely empty: no ``.claude/``, no ``.claude.json``, no + ``.better-memory/``. It is a *subdirectory* of ``tmp_path`` so tests can + park side artifacts (errlog files, seeded repos, ...) in ``tmp_path`` + without polluting the fake home — several scenarios assert the home's + exact file set. + """ + home = tmp_path / "home" + home.mkdir() + return home + + +def run_hook( + module: str, + payload: dict[str, Any] | str | None, + env: dict[str, str], + *, + cwd: Path | str | None = None, + timeout: float = 60, +) -> tuple[int, str, str]: + """Run a hook module as Claude Code does and return (rc, stdout, stderr). + + ``module`` is the dotted module path (e.g. ``better_memory.hooks. + session_bootstrap``). ``payload`` is the hook's stdin: a dict is JSON + encoded, a str is passed verbatim (for malformed-stdin cases), ``None`` + sends empty stdin. ``env`` MUST come from ``isolated_env`` / + ``agentcore_env`` — enforced by the e2e_meta contract test. + """ + if payload is None: + stdin = "" + elif isinstance(payload, str): + stdin = payload + else: + stdin = json.dumps(payload) + proc = subprocess.run( # noqa: S603 — test harness, fixed argv + [sys.executable, "-m", module], + input=stdin, + env=env, + cwd=str(cwd) if cwd is not None else None, + capture_output=True, + text=True, + timeout=timeout, + ) + return proc.returncode, proc.stdout, proc.stderr + + +@asynccontextmanager +async def mcp_session( + env: dict[str, str], + *, + errlog: IO[str] | None = None, + cwd: Path | str | None = None, + initialize: bool = True, + read_timeout: timedelta = DEFAULT_READ_TIMEOUT, + args: tuple[str, ...] = SERVER_ARGS, +) -> AsyncIterator[ClientSession]: + """Drive the real MCP stdio server hermetically. + + * ``env`` MUST explicitly contain ``USERPROFILE`` and ``HOME``: on + Windows the mcp SDK force-inherits the *real* USERPROFILE/APPDATA + underneath ``server.env`` (``{**get_default_environment(), + **server.env}`` in ``mcp/client/stdio/__init__.py``), so omitting them + leaks the real home into the server. Guarded here so a bad env fails + at the call site, not days later as corrupted personal config. + * ``errlog`` must be a real file object with a working ``fileno()`` + (the SDK hands it to subprocess creation as the stderr handle); + ``io.StringIO`` will NOT work. ``None`` inherits pytest's stderr. + * ``cwd`` supports the marker-bridge tests (server resolves the project + dir from its working directory when ``CLAUDE_PROJECT_DIR`` is unset). + * ``initialize=False`` lets negative tests wrap ``session.initialize()`` + in ``pytest.raises(McpError)`` themselves (pre-handshake deaths + surface as McpError code -32000 "Connection closed"). + """ + env_uppers = {k.upper() for k in env} + missing = {"HOME", "USERPROFILE"} - env_uppers + if missing: + raise ValueError( + f"mcp_session env is missing {sorted(missing)} — build it with " + "isolated_env()/agentcore_env(); never hand-roll spawn envs." + ) + if errlog is not None: + errlog.fileno() # raises for StringIO — must be a real OS-level file + + params = StdioServerParameters( + command=sys.executable, + args=list(args), + env=env, + cwd=str(cwd) if cwd is not None else None, + ) + stdio_kwargs: dict[str, Any] = {} + if errlog is not None: + stdio_kwargs["errlog"] = errlog + + async with stdio_client(params, **stdio_kwargs) as (read, write): + async with ClientSession(read, write, read_timeout_seconds=read_timeout) as session: + if initialize: + await session.initialize() + yield session + + +# --------------------------------------------------------------------------- +# real_home_canary — autouse session tripwire (design section 1F) +# --------------------------------------------------------------------------- + +#: Import-time environment snapshot. tests/conftest.py's autouse +#: ``_strip_leaked_claude_env`` deletes BOTH session-id vars before every +#: test, so live-Claude-Code-session detection MUST read the environment at +#: module import (collection) time, before any fixture can strip it. Both +#: var names are covered: Claude Code sets CLAUDE_CODE_SESSION_ID in shell +#: subprocess envs; older harness code reads CLAUDE_SESSION_ID. +_PRISTINE_SESSION_VARS: dict[str, str | None] = { + "CLAUDE_SESSION_ID": os.environ.get("CLAUDE_SESSION_ID"), + "CLAUDE_CODE_SESSION_ID": os.environ.get("CLAUDE_CODE_SESSION_ID"), +} +#: The real home, resolved once at import time from the pristine outer env. +_PRISTINE_REAL_HOME: Path = Path.home() + +#: Canary filename prefix. Files are dot-prefixed, additive-only, and at +#: most two ~36-byte files linger if a run is hard-killed; the next run +#: silently overwrites them (never fails on stale canaries). +CANARY_PREFIX = ".bm-e2e-canary" + + +def _canary_read_json(path: Path) -> Any: + """Parse JSON, folding all failure modes into stable sentinel strings. + + Unreadable-at-setup == unreadable-at-teardown compares equal, so a + malformed real ~/.claude.json can never crash or false-fail the tripwire. + """ + try: + return json.loads(path.read_text(encoding="utf-8")) + except FileNotFoundError: + return "" + except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: + return f"" + + +def _canary_bm_mcp_server(claude_json: Any) -> Any: + """The mcpServers['better-memory'] subtree (semantic, live-session-safe).""" + if not isinstance(claude_json, dict): + return claude_json + servers = claude_json.get("mcpServers") + if not isinstance(servers, dict): + return servers + return servers.get("better-memory") + + +def _canary_bm_hook_entries(settings_json: Any) -> Any: + """Normalized better-memory hook entries per event from settings.json. + + Only entries whose command references better_memory are captured, so a + concurrent live Claude session editing OTHER hooks cannot trip this, + while a leaked install_hooks run (which rewrites the better-memory + entries' command to a tmp venv interpreter, or adds them on a fresh box) + always changes the subtree. + """ + if not isinstance(settings_json, dict): + return settings_json + hooks = settings_json.get("hooks") + if not isinstance(hooks, dict): + return hooks + out: dict[str, list[str]] = {} + for event, groups in hooks.items(): + if not isinstance(groups, list): + continue + entries: list[str] = [] + for group in groups: + if not isinstance(group, dict): + continue + for entry in group.get("hooks", []) or []: + if not isinstance(entry, dict): + continue + command = str(entry.get("command", "")) + if "better_memory" in command or "better-memory" in command: + entries.append(json.dumps(entry, sort_keys=True)) + if entries: + out[str(event)] = sorted(entries) + return out + + +def _canary_entry_names(directory: Path) -> Any: + """Sorted entry names of a real-home dir; canary files excluded.""" + try: + if not directory.is_dir(): + return "" + return sorted(p.name for p in directory.iterdir() if not p.name.startswith(CANARY_PREFIX)) + except OSError as exc: + return f"" + + +def _canary_sha256(path: Path) -> str: + try: + return hashlib.sha256(path.read_bytes()).hexdigest() + except OSError: + return "" + + +def _real_home_snapshot(home: Path) -> dict[str, Any]: + """Tell-tale state that only a leaked e2e subprocess would mutate.""" + claude_json = _canary_read_json(home / ".claude.json") + settings_json = _canary_read_json(home / ".claude" / "settings.json") + return { + "mcp_bm": _canary_bm_mcp_server(claude_json), + "hook_bm": _canary_bm_hook_entries(settings_json), + "install_backups": _canary_entry_names(home / ".better-memory" / "install-backups"), + "skills": _canary_entry_names(home / ".claude" / "skills"), + "claude_json_sha": _canary_sha256(home / ".claude.json"), + "settings_sha": _canary_sha256(home / ".claude" / "settings.json"), + } + + +def _bm_home_env_value(mcp_bm: Any) -> str | None: + if not isinstance(mcp_bm, dict): + return None + env = mcp_bm.get("env") + if not isinstance(env, dict): + return None + value = env.get("BETTER_MEMORY_HOME") + return value if isinstance(value, str) else None + + +def _points_under_tempdir(value: str) -> bool: + try: + return Path(value).resolve().is_relative_to(Path(tempfile.gettempdir()).resolve()) + except (OSError, ValueError): + return False + + +@pytest.fixture(scope="session", autouse=True) +def real_home_canary() -> Iterator[None]: + """Session tripwire: no e2e test may touch the REAL user home. + + Mechanism (judge-fixed spec, design section 1F): + + * Plants two additive dot-canary files (uuid4 content) in the real + ``~/.claude/skills/`` and ``~/.better-memory/`` — only where the + parent ALREADY exists (never creates those dirs on a machine that + lacks them). Filenames carry the xdist worker id so parallel workers + cannot collide; stale canaries from a killed run are overwritten. + * Snapshots semantic tell-tales at setup and compares at teardown: + the parsed mcpServers['better-memory'] subtree, the better-memory + hook entries in settings.json, the install-backups entry set, and + the skills entry-name set. Semantic (not byte-hash) comparison is + live-session-safe: a concurrent Claude Code session rewrites + ~/.claude.json routinely but never rewrites the better-memory + subtrees to a tmp venv, never adds install-backups entries, and + never touches dot-prefixed canaries. + * Smoking gun: mcpServers['better-memory'].env.BETTER_MEMORY_HOME + pointing under tempfile.gettempdir() is a leaked install_hooks run. + * Whole-file hash mismatch WITHOUT a semantic tell-tale is demoted to + a warning (and suppressed entirely inside a live Claude session). + + KNOWN BLIND SPOT (documented per judge round): on a machine where + better-memory is already installed with the same interpreter paths, a + leaked install_hooks run is semantically idempotent and only the + (demoted) hash warning would notice. That case is owned by the + whole-suite meta-run in tests/e2e_meta/test_canary_home.py, which runs + against a fresh seeded canary home where any leak is loud. + """ + home = _PRISTINE_REAL_HOME + worker = os.environ.get("PYTEST_XDIST_WORKER", "main") + canary_name = f"{CANARY_PREFIX}-{worker}" + live_session = any(_PRISTINE_SESSION_VARS.values()) + + planted: dict[Path, str] = {} + for parent in (home / ".claude" / "skills", home / ".better-memory"): + if not parent.is_dir(): + continue # record 'absent' — NEVER create real-home dirs + target = parent / canary_name + content = str(uuid.uuid4()) + try: + target.write_text(content, encoding="utf-8") # overwrite stale + except OSError: + continue # unwritable home (locked-down CI) — skip this canary + planted[target] = content + + before = _real_home_snapshot(home) + + yield + + after = _real_home_snapshot(home) + breaches: list[str] = [] + + for target, content in planted.items(): + try: + actual = target.read_text(encoding="utf-8") + except OSError: + breaches.append(f"canary file deleted or unreadable: {target}") + continue + if actual != content: + breaches.append(f"canary file overwritten: {target}") + + for key, label in ( + ("mcp_bm", "~/.claude.json mcpServers['better-memory']"), + ("hook_bm", "~/.claude/settings.json better-memory hook entries"), + ("install_backups", "~/.better-memory/install-backups entries"), + ("skills", "~/.claude/skills entry names"), + ): + if before[key] != after[key]: + breaches.append( + f"{label} changed during the e2e session:\n" + f" setup: {before[key]!r}\n" + f" teardown: {after[key]!r}" + ) + + # Smoking gun: a tmp-path BETTER_MEMORY_HOME in the real config. + after_bm_home = _bm_home_env_value(after["mcp_bm"]) + if after_bm_home is not None and _points_under_tempdir(after_bm_home): + message = ( + "real ~/.claude.json mcpServers['better-memory'].env.BETTER_MEMORY_HOME " + f"points under tempfile.gettempdir(): {after_bm_home!r}" + ) + if after_bm_home == _bm_home_env_value(before["mcp_bm"]): + # Pre-existing damage from an earlier leak — this run did not do + # it; warn instead of permanently bricking every suite run. + warnings.warn( + f"PRE-EXISTING real-home damage (not caused by this run): {message}", + stacklevel=1, + ) + else: + breaches.append(f"SMOKING GUN (leaked install_hooks run): {message}") + + # Byte-hash drift with NO semantic tell-tale: warning only (a live + # Claude session rewrites these files routinely — suppress there). + hash_drift = [ + name + for name, key in ((".claude.json", "claude_json_sha"), ("settings.json", "settings_sha")) + if before[key] != after[key] + ] + if hash_drift and not breaches and not live_session: + warnings.warn( + "real-home file bytes changed without a semantic better-memory " + f"tell-tale: {hash_drift} — not treated as a breach (see fixture " + "docstring); investigate if it repeats on quiescent machines.", + stacklevel=1, + ) + + for target in planted: + try: + target.unlink(missing_ok=True) + except OSError: + pass + + if breaches: + pytest.fail( + "HARNESS ISOLATION BREACH — an e2e test touched the REAL user " + "home. Every subprocess env must come from isolated_env()/" + "agentcore_env(); details:\n" + "\n".join(breaches), + pytrace=False, + ) diff --git a/tests/e2e/test_agentcore_neg.py b/tests/e2e/test_agentcore_neg.py new file mode 100644 index 0000000..f9da9e6 --- /dev/null +++ b/tests/e2e/test_agentcore_neg.py @@ -0,0 +1,322 @@ +"""T2 agentcore negative / misconfiguration scenarios D8-D11 (design 1D). + +The failure surfaces a brand-new agentcore user actually hits: + +* D8 ``e2e-ac-neg-prehandshake-config-errors`` — parametrized idvar-gate + (KNOWN-DEFECT PIN) + backend-name typo, sharing one raw-spawn helper; +* D9 ``e2e-ac-neg-missing-agentcore-json`` — raw + SDK-client levels + (the enduring negative that survives the idvar-gate fix); +* D10 ``e2e-ac-neg-corrupt-agentcore-json`` — truncated JSON + forward + schema_version, both fail loudly, neither offers remediation (pin); +* D11 ``e2e-ac-neg-boto3-missing`` — PYTHONPATH shadow simulating a plain + ``pip install better-memory`` (no [agentcore] extra), raw traceback + with no install hint (PRODUCT GAP PIN), content-based control run. +""" + +from __future__ import annotations + +import json +import sqlite3 +from contextlib import closing +from datetime import timedelta +from pathlib import Path + +import pytest +from mcp.shared.exceptions import McpError + +from tests.e2e._agentcore_env import ( + agentcore_env, + remove_dummy_id_vars_pins, + write_fake_agentcore_json, +) +from tests.e2e._env import isolated_env +from tests.e2e._fake_agentcore import FakeAgentCore +from tests.e2e.conftest import mcp_session, run_hook + +_SERVER_MODULE = "better_memory.mcp" + + +def _bm_home(home: Path) -> Path: + return home / ".better-memory" + + +def _table_count(db_path: Path) -> int: + with closing(sqlite3.connect(db_path)) as conn: + (count,) = conn.execute( + "SELECT COUNT(*) FROM sqlite_master WHERE type='table'" + ).fetchone() + return int(count) + + +def _assert_prehandshake_death( + rc: int, out: str, err: str, expected_stderr: list[str] +) -> None: + """Shared oracle for a pre-JSON-RPC server death. + + ``stdout == ''`` is the load-bearing half: zero bytes means the death + is strictly pre-handshake — Claude Code sees an opaque spawn failure, + never a tool error.""" + assert rc == 1 + assert out == "" + for substring in expected_stderr: + assert substring in err, f"{substring!r} not in stderr:\n{err}" + + +# --------------------------------------------------------------------------- +# D8 — pre-handshake config errors (parametrized) +# --------------------------------------------------------------------------- + + +class TestPrehandshakeConfigErrors: + @pytest.mark.parametrize("case", ["idvar-gate", "backend-typo"]) + def test_config_error_kills_server_prehandshake_before_any_disk_write( + self, case: str, clean_slate_home: Path + ) -> None: + """Two get_config() raises sharing one spawn helper + one shared + ordering pin (config is validated before ANY disk write — no + memory.db afterwards, distinguishing config-stage death from the + post-config failures in TestMissingAgentCoreJson). + + [idvar-gate] KNOWN-DEFECT PIN (design section 4 item 1): the + presence-only gate at better_memory/config.py:293-301 is vestigial + — its values are consumed by nothing (IDs come from + agentcore.json; commit 0056935 switched the transport) and its own + remediation text ('agentcore init') cannot clear it, since init + never sets env vars. Every documented agentcore setup dies + pre-handshake like this. DELETE this case together with the + FIXME(idvar-gate) dummy vars in tests/e2e/_agentcore_env.py when + the product fix lands. + + [backend-typo] 'AgentCore' (case typo) must die loudly with the + offending value echoed — never silently coerce/default to sqlite + (the worst-case silent misconfig: the user believes they are on + AWS while every memory lands in a local file).""" + bm_home = _bm_home(clean_slate_home) + + if case == "idvar-gate": + with FakeAgentCore() as fake: + write_fake_agentcore_json(bm_home) # valid json — gate still kills + env = agentcore_env( + clean_slate_home, fake.port, **remove_dummy_id_vars_pins() + ) + rc, out, err = run_hook(_SERVER_MODULE, None, env) + assert fake.requests == [] + _assert_prehandshake_death( + rc, + out, + err, + [ + "BETTER_MEMORY_STORAGE_BACKEND=agentcore requires both", + "BETTER_MEMORY_AGENTCORE_SEMANTIC_MEMORY_ID", + "BETTER_MEMORY_AGENTCORE_EPISODIC_MEMORY_ID", + # Pins the remediation that cannot actually clear the + # gate — anyone rewording it re-reads this docstring. + "agentcore init", + ], + ) + else: + env = isolated_env( + clean_slate_home, BETTER_MEMORY_STORAGE_BACKEND="AgentCore" + ) + rc, out, err = run_hook(_SERVER_MODULE, None, env) + _assert_prehandshake_death( + rc, + out, + err, + [ + "ValueError", + "is not one of", + "'AgentCore'", # repr'd — the user can SEE the typo + ], + ) + + # Shared ordering pin: get_config raises before connect() — + # config failures leave zero disk artifacts. + assert not (bm_home / "memory.db").exists() + + +# --------------------------------------------------------------------------- +# D9 — missing agentcore.json (the enduring negative) +# --------------------------------------------------------------------------- + + +class TestMissingAgentCoreJson: + def test_raw_spawn_remediation_text_and_migrated_before_failure( + self, clean_slate_home: Path + ) -> None: + """ID vars set but no agentcore.json (the classic second-machine + setup failure): pre-handshake FileNotFoundError with the one + accurate remediation hint in this flow. Plus the ordering/docs + pin: create_server migrates BOTH sqlite DBs BEFORE build_backend + fails, so a *migrated* memory.db exists after the failed boot + (sqlite_master > 0 — distinguishing it from the hook's schema-less + artifact, and pinning the false 'No SQLite traffic' docs claim, + design section 4 item 12).""" + bm_home = _bm_home(clean_slate_home) + with FakeAgentCore() as fake: + env = agentcore_env(clean_slate_home, fake.port) + rc, out, err = run_hook(_SERVER_MODULE, None, env) + assert fake.requests == [] + + _assert_prehandshake_death( + rc, + out, + err, + [ + "FileNotFoundError", + "agentcore.json not found", + "better-memory agentcore init", + ], + ) + assert (bm_home / "memory.db").exists() + assert _table_count(bm_home / "memory.db") > 0 + + async def test_sdk_client_sees_mcperror_connection_closed( + self, clean_slate_home: Path, tmp_path: Path + ) -> None: + """What Claude Code's client actually experiences: initialize() + raises McpError code -32000 'Connection closed'; the real + diagnostic is only recoverable via the errlog channel (a REAL file + — the SDK hands it to subprocess creation as the stderr handle). + Never assert an exit code here: the SDK never exposes the Process. + McpError is caught TIGHTLY around initialize() (inside both + contexts) — letting it escape arrives double-wrapped in + ExceptionGroups.""" + errlog_path = tmp_path / "server_stderr.txt" + with FakeAgentCore() as fake: + env = agentcore_env(clean_slate_home, fake.port) + with errlog_path.open("w", encoding="utf-8") as errlog: + async with mcp_session( + env, + errlog=errlog, + initialize=False, + # Backstop for the theoretical registration-after-EOF + # race: worst case degrades to a McpError timeout, so + # pytest.raises(McpError) still holds. + read_timeout=timedelta(seconds=20), + ) as session: + with pytest.raises(McpError) as exc_info: + await session.initialize() + # Both context managers exit cleanly — the process is + # already dead, so shutdown returns immediately. + + assert exc_info.value.error.code == -32000 + assert "Connection closed" in exc_info.value.error.message + assert "agentcore.json not found" in errlog_path.read_text( + encoding="utf-8" + ) + + +# --------------------------------------------------------------------------- +# D10 — corrupt agentcore.json +# --------------------------------------------------------------------------- + + +class TestCorruptAgentCoreJson: + @pytest.mark.parametrize( + ("case", "case_stderr"), + [ + ("truncated", ["failed to parse"]), + ("schema-v2", ["unsupported schema_version=2", "expected 1"]), + ], + ) + def test_corrupt_json_dies_loudly_naming_the_file( + self, case: str, case_stderr: list[str], clean_slate_home: Path + ) -> None: + """Truncated JSON (killed-mid-write / hand-edit) and a forward + schema_version both raise AgentCoreConfigError naming the file — + the fail-loud contract the persistence module docstring promises. + NEGATIVE PIN (design section 4 item 8): unlike the missing-file + path, the corrupt-file error offers NO 'agentcore init' + remediation — a flagged product gap; flip the last assertion when + remediation text is added.""" + bm_home = _bm_home(clean_slate_home) + if case == "truncated": + bm_home.mkdir(parents=True) + (bm_home / "agentcore.json").write_text( + '{"schema_version": 1, "region"', encoding="utf-8" + ) + else: + write_fake_agentcore_json(bm_home) + raw = json.loads( + (bm_home / "agentcore.json").read_text(encoding="utf-8") + ) + raw["schema_version"] = 2 + (bm_home / "agentcore.json").write_text( + json.dumps(raw), encoding="utf-8" + ) + + with FakeAgentCore() as fake: + env = agentcore_env(clean_slate_home, fake.port) + rc, out, err = run_hook(_SERVER_MODULE, None, env) + assert fake.requests == [] + + _assert_prehandshake_death( + rc, + out, + err, + ["AgentCoreConfigError", "agentcore.json", *case_stderr], + ) + # Product-gap pin: no remediation breadcrumb on the corrupt path. + assert "agentcore init" not in err + + +# --------------------------------------------------------------------------- +# D11 — boto3 missing (plain install without the [agentcore] extra) +# --------------------------------------------------------------------------- + + +class TestBoto3MissingImportSurface: + def test_shadowed_boto3_yields_raw_traceback_without_install_hint( + self, clean_slate_home: Path, tmp_path: Path + ) -> None: + """PRODUCT GAP PIN (design section 4 item 7): a plain + ``pip install better-memory`` (no [agentcore] extra) + + backend=agentcore dies at the lazy ``import boto3`` in + better_memory/storage/factory.py with a RAW ModuleNotFoundError — + no 'better-memory[agentcore]' / 'pip install' breadcrumb. INVERT + the two 'not in' assertions when the friendly message lands. + + boto3 is a dev-group dep in this venv, so absence is simulated + with a PYTHONPATH-front shadow module. The shadow's raise SITE + differs from a genuinely absent package (module body vs import + machinery), but the user-visible class + message are identical — + assertions use only those, never traceback frames. + + The control run (same env minus the shadow) is CONTENT-based — + never an rc comparison: the control server boots fully and its + exit code on stdin EOF is SDK/anyio-dependent.""" + bm_home = _bm_home(clean_slate_home) + shadow_dir = tmp_path / "shadow" + shadow_dir.mkdir() + (shadow_dir / "boto3.py").write_text( + "raise ModuleNotFoundError(\"No module named 'boto3'\")\n", + encoding="utf-8", + ) + + with FakeAgentCore() as fake: + write_fake_agentcore_json(bm_home) + env = agentcore_env( + clean_slate_home, fake.port, PYTHONPATH=str(shadow_dir) + ) + rc, out, err = run_hook(_SERVER_MODULE, None, env) + assert fake.requests == [] + + _assert_prehandshake_death( + rc, + out, + err, + ["ModuleNotFoundError", "No module named 'boto3'"], + ) + # PRODUCT GAP PIN — inverted assertions (see docstring). + assert "better-memory[agentcore]" not in err + assert "pip install" not in err + + # Vacuity guard: the same env WITHOUT the shadow must not die + # on the boto3 import — proving the shadow (not some other + # misconfig) caused the failure above. Also catches boto3 + # becoming a top-level import reachable from module import. + control_env = agentcore_env(clean_slate_home, fake.port) + _rc, _out, control_err = run_hook(_SERVER_MODULE, None, control_env) + assert "ModuleNotFoundError" not in control_err + assert "No module named 'boto3'" not in control_err diff --git a/tests/e2e/test_agentcore_t2.py b/tests/e2e/test_agentcore_t2.py new file mode 100644 index 0000000..c1d52c8 --- /dev/null +++ b/tests/e2e/test_agentcore_t2.py @@ -0,0 +1,850 @@ +"""T2 hermetic agentcore scenarios D1-D7 (design section 1D). + +Every test runs against the local :class:`FakeAgentCore` endpoint via the +``AWS_ENDPOINT_URL`` seam — no mocks on the wire path: real botocore +serialization, real SigV4 signing (with fake creds), real HTTP. + +Covers: + +* D1 ``e2e-ac-server-boot-tools-hidden`` +* D2 ``e2e-ac-retrieve-polarity-fanout`` +* D3 ``e2e-ac-mcp-dispatch-gap-pin`` (KNOWN-DEFECT PIN) +* D4 ``e2e-ac-backend-wire-fidelity`` (in-process backend, real boto3) +* D5 ``e2e-ac-session-close-closure-and-env-gate`` +* D6 ``e2e-ac-contextual-inject-wire-and-degradation`` +* D7 ``e2e-ac-region-split-brain-pin`` (KNOWN-DEFECT PIN) +""" + +from __future__ import annotations + +import hashlib +import json +import sqlite3 +from contextlib import closing +from pathlib import Path +from typing import Any + +import pytest + +from tests.e2e._agentcore_env import ( + DUMMY_EPISODIC_MEMORY_ID, + DUMMY_ID_VARS, + agentcore_env, + remove_dummy_id_vars_pins, + write_fake_agentcore_json, +) +from tests.e2e._fake_agentcore import FakeAgentCore +from tests.e2e.conftest import mcp_session, run_hook + +EXPECTED_TOOL_SUBSET = { + "memory.observe", + "memory.retrieve", + "memory.semantic_observe", + "memory.session_bootstrap", + "memory.record_use", +} +SYNTHESIZE_TOOLS = { + "memory.synthesize_next_get_context", + "memory.synthesize_next_apply", +} + + +def _bm_home(home: Path) -> Path: + return home / ".better-memory" + + +def _table_names(db_path: Path) -> set[str]: + with closing(sqlite3.connect(db_path)) as conn: + rows = conn.execute( + "SELECT name FROM sqlite_master WHERE type='table'" + ).fetchall() + return {row[0] for row in rows} + + +def _tool_text(result: Any) -> str: + return result.content[0].text + + +def _polarity_filter_value(body: dict) -> str | None: + for entry in body.get("metadataFilters", []): + if entry.get("left", {}).get("metadataKey") == "polarity": + return entry["right"]["metadataValue"]["stringValue"] + return None + + +def _has_active_status_filter(body: dict) -> bool: + return any( + entry.get("left", {}).get("metadataKey") == "status" + and entry.get("operator") == "EQUALS_TO" + and entry.get("right", {}).get("metadataValue", {}).get("stringValue") + == "active" + for entry in body.get("metadataFilters", []) + ) + + +# --------------------------------------------------------------------------- +# D1 — server boot, tools hidden, sqlite still migrated +# --------------------------------------------------------------------------- + + +class TestServerBootToolsHidden: + async def test_boot_hides_synthesize_tools_and_still_migrates_sqlite( + self, clean_slate_home: Path + ) -> None: + """Real server boots from a fabricated agentcore.json against the + fake endpoint. Boot makes ZERO AWS calls; synthesize tools are + hidden (supports_synthesis=False); and — docs-contradiction pin — + both local sqlite DBs are still created + migrated in agentcore + mode (design section 4 item 12: the 'No SQLite traffic' doc claim + is false).""" + bm_home = _bm_home(clean_slate_home) + with FakeAgentCore() as fake: + write_fake_agentcore_json(bm_home) + env = agentcore_env(clean_slate_home, fake.port) + + async with mcp_session(env) as session: + tools = await session.list_tools() + + names = {tool.name for tool in tools.tools} + assert EXPECTED_TOOL_SUBSET <= names + assert not (SYNTHESIZE_TOOLS & names) + # Boot never touches AWS (boto3 client construction only). + assert fake.requests == [] + + tables = _table_names(bm_home / "memory.db") + # Subset, never the exact column/table set (design rule). + assert {"observations", "episodes", "hook_errors"} <= tables + assert (bm_home / "knowledge.db").exists() + + +# --------------------------------------------------------------------------- +# D2 — retrieve polarity fan-out (the flagship wired path) +# --------------------------------------------------------------------------- + + +class TestRetrievePolarityFanout: + async def test_retrieve_fans_out_three_filtered_list_records( + self, clean_slate_home: Path + ) -> None: + """``memory.retrieve`` is the ONE data tool wired to + ``AgentCoreBackend`` (ReflectionToolHandlers → backend.retrieve): + exactly 3 ListMemoryRecords against the EPISODIC memory's + reflections namespace, order-insensitive polarity set + {do,dont,neutral}, each carrying a status=active filter; buckets + come back as exact empty lists. A single-polarity call restricts + the fan-out to exactly 1 request.""" + bm_home = _bm_home(clean_slate_home) + with FakeAgentCore() as fake: + write_fake_agentcore_json(bm_home) + fake.set_response("ListMemoryRecords", {"memoryRecordSummaries": []}) + env = agentcore_env(clean_slate_home, fake.port) + + async with mcp_session(env) as session: + result = await session.call_tool("memory.retrieve", {}) + assert not result.isError + buckets = json.loads(_tool_text(result)) + assert {"do", "dont", "neutral"} <= set(buckets) + # Exact empty lists — the clean-slate contract a brand-new + # agentcore user sees. + assert buckets["do"] == [] + assert buckets["dont"] == [] + assert buckets["neutral"] == [] + + requests = list(fake.requests) + # Exactly 3 wire requests, all ListMemoryRecords — a + # collapsed/unfiltered rewrite flips this red. + assert len(requests) == 3 + assert {r.operation for r in requests} == {"ListMemoryRecords"} + for request in requests: + assert "EPI-FAKE-0001" in request.path + assert "reflections" in request.body.get("namespace", "") + assert _has_active_status_filter(request.body), request.body + polarities = { + _polarity_filter_value(r.body) for r in requests + } + # Order-insensitive: ThreadPoolExecutor fan-out is unordered. + assert polarities == {"do", "dont", "neutral"} + + # Single-polarity leg: exactly one request, filter 'do'. + fake.clear() + result2 = await session.call_tool( + "memory.retrieve", {"polarity": "do"} + ) + assert not result2.isError + single = fake.requests_for("ListMemoryRecords") + assert len(single) == 1 + assert _polarity_filter_value(single[0].body) == "do" + + +# --------------------------------------------------------------------------- +# D3 — MCP dispatch gap (KNOWN-DEFECT PIN) +# --------------------------------------------------------------------------- + + +class TestMcpDispatchGapPin: + async def test_observe_semantic_record_use_never_reach_the_wire( + self, clean_slate_home: Path + ) -> None: + """KNOWN-DEFECT PIN (design section 4 item 2, the highest-priority + product finding): in agentcore mode ``memory.observe``, + ``memory.semantic_observe`` and ``memory.record_use`` dispatch to + LOCAL sqlite services, never to AgentCoreBackend — the registry in + better_memory/mcp/server.py:249-271 constructs + ObservationToolHandlers/SemanticToolHandlers on sqlite services + unconditionally. Agentcore users' memories silently land in a + local file. + + DELETE (and replace with wire tests promoted from + TestBackendWireFidelity) the day the dispatch layer is wired to + the backend — this test flips loudly on that day.""" + bm_home = _bm_home(clean_slate_home) + with FakeAgentCore() as fake: + write_fake_agentcore_json(bm_home) + env = agentcore_env(clean_slate_home, fake.port) + + async with mcp_session(env) as session: + observe = await session.call_tool( + "memory.observe", {"content": "dispatch-gap-marker-obs"} + ) + assert not observe.isError + obs_id = json.loads(_tool_text(observe))["id"] + # A local uuid, not an AgentCore eventId from the fake. + assert obs_id + + semantic = await session.call_tool( + "memory.semantic_observe", + {"content": "dispatch-gap-marker-semantic"}, + ) + assert not semantic.isError + + record_use = await session.call_tool( + "memory.record_use", {"id": obs_id, "outcome": "success"} + ) + assert not record_use.isError + assert json.loads(_tool_text(record_use)) == {"ok": True} + + # THE pin: zero wire requests across all three data tools. + assert fake.requests == [] + + # ... and the writes landed in the local sqlite file instead. + with closing(sqlite3.connect(_bm_home(clean_slate_home) / "memory.db")) as conn: + obs_rows = conn.execute( + "SELECT content FROM observations WHERE content = ?", + ("dispatch-gap-marker-obs",), + ).fetchall() + sem_rows = conn.execute( + "SELECT content FROM semantic_memories WHERE content = ?", + ("dispatch-gap-marker-semantic",), + ).fetchall() + assert len(obs_rows) == 1 + assert len(sem_rows) == 1 + + +# --------------------------------------------------------------------------- +# D4 — backend wire fidelity (in-process AgentCoreBackend, real boto3) +# --------------------------------------------------------------------------- + + +@pytest.fixture +def scrubbed_aws_process_env( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + """In-process boto3 clients read THIS process's env for the config / + credential chains. Explicit kwargs (endpoint_url, creds, region) win + for everything we assert, but scrub the hazardous vars anyway so a dev + shell with AWS_PROFILE/SSO material can never influence these tests.""" + for var in ( + "AWS_PROFILE", + "AWS_DEFAULT_REGION", + "AWS_REGION", + "AWS_SESSION_TOKEN", + "AWS_ENDPOINT_URL", + "AWS_IGNORE_CONFIGURED_ENDPOINT_URLS", + ): + monkeypatch.delenv(var, raising=False) + monkeypatch.setenv( + "AWS_SHARED_CREDENTIALS_FILE", str(tmp_path / "no-such-credentials") + ) + monkeypatch.setenv("AWS_CONFIG_FILE", str(tmp_path / "no-such-config")) + + +#: Real botocore request validation enforces a 40-char minimum on +#: memoryRecordId — short ids like 'refl-1' are rejected client-side +#: before any wire traffic (a wire-fidelity finding in itself). +_REFL_RECORD_ID = "refl-e2e-" + "0" * 30 + "1" +_SEM_RECORD_ID = "sem-rec-e2e-" + "1" * 30 + + +@pytest.mark.usefixtures("scrubbed_aws_process_env") +class TestBackendWireFidelity: + """Direct AgentCoreBackend with REAL boto3 clients against the fake — + real botocore serialization, no MagicMocks. Re-homed here from the MCP + layer per the judges: these paths are unreachable over MCP today (see + TestMcpDispatchGapPin), but hooks (contextual_inject, session_close) + and the future dispatch wiring hit them for real.""" + + def _make_backend( + self, + fake: FakeAgentCore, + tmp_path: Path, + *, + session_id: str | None = "e2e-wire-session", + project: str = "e2e-project", + ) -> Any: + import boto3 + from botocore.config import Config as BotoConfig + + from better_memory.storage.agentcore import AgentCoreBackend + + cfg = write_fake_agentcore_json(tmp_path / ".better-memory") + client_kwargs: dict[str, Any] = { + "endpoint_url": fake.endpoint_url, + "region_name": cfg.region, + "aws_access_key_id": "bm-e2e-fake", + "aws_secret_access_key": "bm-e2e-fake-secret", + "config": BotoConfig(retries={"mode": "standard", "max_attempts": 5}), + } + return AgentCoreBackend( + config=cfg, + data_client=boto3.client("bedrock-agentcore", **client_kwargs), + control_client=boto3.client( + "bedrock-agentcore-control", **client_kwargs + ), + session_id=session_id, + project=project, + ) + + async def test_observe_create_event_wire_shape(self, tmp_path: Path) -> None: + """CreateEvent: EPI routing, sessionId/actorId from construction, + USER role, stringValue-ONLY event metadata, None values dropped.""" + with FakeAgentCore() as fake: + fake.set_response("CreateEvent", {"event": {"eventId": "fake-evt-1"}}) + backend = self._make_backend(fake, tmp_path) + + event_id = await backend.observe( + content="wire-marker-observe-7f3a", + outcome="failure", + component="e2ecomp", + theme="gotcha", + ) + assert event_id == "fake-evt-1" + + requests = fake.requests_for("CreateEvent") + assert len(requests) == 1 + request = requests[0] + assert "EPI-FAKE-0001" in request.path + body = request.body + assert body["sessionId"] == "e2e-wire-session" + assert body["actorId"] == "e2e-project" + conversational = body["payload"][0]["conversational"] + assert conversational["role"] == "USER" + assert conversational["content"]["text"] == "wire-marker-observe-7f3a" + metadata = body["metadata"] + assert metadata["outcome"] == {"stringValue": "failure"} + assert metadata["component"] == {"stringValue": "e2ecomp"} + assert metadata["theme"] == {"stringValue": "gotcha"} + # None-valued fields never serialize (real AWS rejects nulls). + assert "scope_path" not in metadata + assert "trigger_type" not in metadata + # Event metadata is stringValue-only (numberValue on events is + # rejected by real AWS). + for value in metadata.values(): + assert set(value) == {"stringValue"} + + async def test_observe_without_session_id_raises_before_wire( + self, tmp_path: Path + ) -> None: + """observe with no session id → ValueError, ZERO wire requests + (no uuid4 fallback fabricating identities — folded set-3 gap-2).""" + with FakeAgentCore() as fake: + backend = self._make_backend(fake, tmp_path, session_id=None) + with pytest.raises(ValueError, match="requires session_id"): + await backend.observe(content="never-sent") + assert fake.requests == [] + + def test_semantic_observe_sha256_request_identifier_and_routing( + self, tmp_path: Path + ) -> None: + """BatchCreateMemoryRecords: requestIdentifier is EXACTLY + sha256(content)[:80] (computed independently here — a genuine + oracle, not an echo), SEM routing (never EPI), strategy id flows + from agentcore.json, initial counter metadata present.""" + content = "prefers uv over pip for this repo — semantic-marker-91c2" + expected_req_id = hashlib.sha256(content.encode("utf-8")).hexdigest()[:80] + + with FakeAgentCore() as fake: + fake.set_response( + "BatchCreateMemoryRecords", + { + "successfulRecords": [{"memoryRecordId": "fake-rec-77"}], + "failedRecords": [], + }, + ) + backend = self._make_backend(fake, tmp_path) + + record_id = backend.semantic_observe(content=content) + assert record_id == "fake-rec-77" + + requests = fake.requests_for("BatchCreateMemoryRecords") + assert len(requests) == 1 + request = requests[0] + assert "SEM-FAKE-0001" in request.path + assert "EPI-FAKE-0001" not in request.text() + record = request.body["records"][0] + assert record["requestIdentifier"] == expected_req_id + namespaces = record["namespaces"] + assert len(namespaces) == 1 + assert "e2e-project" in namespaces[0] + assert "semantic" in namespaces[0] + assert record["memoryStrategyId"] == "STRAT-SEM-1" + metadata = record["metadata"] + assert metadata["status"] == {"stringValue": "active"} + assert metadata["useful_count"] == {"numberValue": 0} + + def test_record_use_strips_system_metadata_full_snapshot( + self, tmp_path: Path + ) -> None: + """record_use sends a FULL metadata snapshot with every + x-amz-agentcore-memory-* key STRIPPED (echoing them back is a real + AWS 400 'reserved names or prefixes' — the live smoke already hit + it once) and the counter incremented from the GET response.""" + with FakeAgentCore() as fake: + fake.set_response( + "GetMemoryRecord", + { + "memoryRecord": { + "memoryRecordId": _REFL_RECORD_ID, + "metadata": { + "useful_count": {"numberValue": 2}, + "status": {"stringValue": "active"}, + "x-amz-agentcore-memory-createdAt": { + "stringValue": "2026-01-01T00:00:00Z" + }, + "x-amz-agentcore-memory-recordType": { + "stringValue": "EXTRACTED" + }, + }, + } + }, + ) + fake.set_response( + "BatchUpdateMemoryRecords", + { + "successfulRecords": [{"memoryRecordId": _REFL_RECORD_ID}], + "failedRecords": [], + }, + ) + backend = self._make_backend(fake, tmp_path) + + backend.record_use(_REFL_RECORD_ID, outcome="success") + + gets = fake.requests_for("GetMemoryRecord") + updates = fake.requests_for("BatchUpdateMemoryRecords") + assert len(gets) == 1 + assert len(updates) == 1 + assert "EPI-FAKE-0001" in gets[0].path + assert "EPI-FAKE-0001" in updates[0].path + + metadata = updates[0].body["records"][0]["metadata"] + assert not any( + key.startswith("x-amz-agentcore-memory-") for key in metadata + ), metadata + assert metadata["useful_count"]["numberValue"] == pytest.approx(3) + # Full snapshot, not a diff: non-counter keys preserved. + assert metadata["status"] == {"stringValue": "active"} + assert "last_credited_at" in metadata # presence only + + def test_credit_one_semantic_kind_routes_to_sem_and_strips( + self, tmp_path: Path + ) -> None: + """credit_one duplicates the snapshot path record_use uses and is + what the rating flow actually hits — same strip contract, plus + kind='semantic' must route lookup AND update to the SEMANTIC + memory (an episodic lookup would 404 on real AWS).""" + with FakeAgentCore() as fake: + fake.set_response( + "GetMemoryRecord", + { + "memoryRecord": { + "memoryRecordId": _SEM_RECORD_ID, + "metadata": { + "useful_count": {"numberValue": 4}, + "status": {"stringValue": "active"}, + "x-amz-agentcore-memory-updatedAt": { + "stringValue": "2026-01-02T00:00:00Z" + }, + }, + } + }, + ) + fake.set_response( + "BatchUpdateMemoryRecords", + { + "successfulRecords": [{"memoryRecordId": _SEM_RECORD_ID}], + "failedRecords": [], + }, + ) + backend = self._make_backend(fake, tmp_path) + + result = backend.credit_one( + session_id="e2e-wire-session", + kind="semantic", + id=_SEM_RECORD_ID, + classification="cited", + ) + assert result == {"applied": _SEM_RECORD_ID, "skipped": None} + + gets = fake.requests_for("GetMemoryRecord") + updates = fake.requests_for("BatchUpdateMemoryRecords") + assert len(gets) == 1 + assert len(updates) == 1 + assert "SEM-FAKE-0001" in gets[0].path + assert "SEM-FAKE-0001" in updates[0].path + + metadata = updates[0].body["records"][0]["metadata"] + assert not any( + key.startswith("x-amz-agentcore-memory-") for key in metadata + ), metadata + # cited → useful_count += 1 (4 → 5). + assert metadata["useful_count"]["numberValue"] == pytest.approx(5) + assert "last_credited_at" in metadata + + +# --------------------------------------------------------------------------- +# D5 — session_close closure event + env gate +# --------------------------------------------------------------------------- + + +class TestSessionCloseClosureAndEnvGate: + def test_stop_hook_fires_one_closure_event_signed_with_json_region( + self, clean_slate_home: Path, tmp_path: Path + ) -> None: + """Case A: the Stop hook builds a REAL boto3 client (zero coverage + elsewhere — the hook unit tests MagicMock it) and fires exactly one + role=OTHER closure CreateEvent, SigV4-signed with agentcore.json's + region (json says us-east-1 while the env pin stays eu-west-2 — + proving cfg.region, not env, drives the hook client); the spool + marker is still written.""" + bm_home = _bm_home(clean_slate_home) + with FakeAgentCore() as fake: + # Deliberately different from the env region pin (eu-west-2). + write_fake_agentcore_json(bm_home, region="us-east-1") + env = agentcore_env(clean_slate_home, fake.port) + + rc, out, err = run_hook( + "better_memory.hooks.session_close", + { + "session_id": "payload-session-ignored", + "cwd": str(tmp_path / "proj"), + "hook_event_name": "Stop", + }, + env, + ) + assert rc == 0 + # No memory.db → the rating-directive branch short-circuits → + # empty stdout is the strong no-block oracle. + assert out == "" + assert "Traceback" not in err + + requests = list(fake.requests) + assert len(requests) == 1 + request = requests[0] + assert request.operation == "CreateEvent" + assert "EPI-FAKE-0001" in request.path + conversational = request.body["payload"][0]["conversational"] + assert conversational["role"] == "OTHER" + # env CLAUDE_SESSION_ID wins over the stdin payload session_id. + assert request.body["sessionId"] == "e2e-session-1" + # The hook signs with agentcore.json's region (session_close.py + # builds its client from cfg.region) — the other half of the + # region split-brain pinned in TestRegionSplitBrainPin. + assert request.sigv4_region == "us-east-1" + + markers = list((bm_home / "spool").glob("*_session_end_*.json")) + assert len(markers) == 1 + marker_body = json.loads(markers[0].read_text(encoding="utf-8")) + assert marker_body["event_type"] == "session_end" + # Closure succeeded → no hook_errors write → no memory.db at all. + assert not (bm_home / "memory.db").exists() + + def test_stop_hook_without_backend_env_skips_aws_but_writes_marker( + self, clean_slate_home: Path, tmp_path: Path + ) -> None: + """Case B — KNOWN-DEFECT PIN (design section 4 item 5, hook + env-propagation gap): the installer writes NO env into hook + commands, so a real agentcore user's Stop hook runs WITHOUT + BETTER_MEMORY_STORAGE_BACKEND and the closure event silently never + fires (session_close.py's guard reads the raw env and returns + before its try block — no hook_errors row, no error, nothing). + The spool marker is still written. Flips loudly when the installer + starts propagating hook env (the intended fix) or the guard + condition changes.""" + bm_home = _bm_home(clean_slate_home) + with FakeAgentCore() as fake: + write_fake_agentcore_json(bm_home) + env = agentcore_env( + clean_slate_home, fake.port, BETTER_MEMORY_STORAGE_BACKEND=None + ) + + rc, out, err = run_hook( + "better_memory.hooks.session_close", + { + "session_id": "e2e-session-1", + "cwd": str(tmp_path / "proj"), + "hook_event_name": "Stop", + }, + env, + ) + assert rc == 0 + assert out == "" + assert "Traceback" not in err + # THE pin: zero wire requests — silent skip. + assert fake.requests == [] + + markers = list((bm_home / "spool").glob("*_session_end_*.json")) + assert len(markers) == 1 + # Guard short-circuits BEFORE the try block: no record_hook_error, + # hence no memory.db (and therefore no hook_errors row) at all. + assert not (bm_home / "memory.db").exists() + + +# --------------------------------------------------------------------------- +# D6 — contextual_inject wire + degradation (the only shipped per-prompt +# backend path) +# --------------------------------------------------------------------------- + +_INJECT_PROMPT = "how do I deploy with docker compose" + +#: A reflection record whose title clears the min-hits floor (3 distinct +#: whole-word keyword hits from the prompt: deploy/docker/compose >= the +#: default BETTER_MEMORY_CONTEXT_MIN_HITS=2). +_DOCKER_REFLECTION_SUMMARY = { + "memoryRecordId": "refl-fake-docker", + "content": { + "text": json.dumps( + { + "title": "docker compose deploy pitfalls", + "phase": "implementation", + "use_cases": "deploying docker compose stacks", + "hints": "- always pass --build", + "confidence": 0.9, + } + ) + }, + "metadata": { + "useful_count": {"numberValue": 2}, + "status": {"stringValue": "active"}, + }, +} + + +def _inject_payload(tmp_path: Path) -> dict[str, Any]: + return { + "hook_event_name": "UserPromptSubmit", + "prompt": _INJECT_PROMPT, + "session_id": "e2e-inject-session", + "cwd": str(tmp_path / "proj"), + } + + +class TestContextualInjectWireAndDegradation: + def test_case_a_happy_path_hits_both_memories_and_injects( + self, clean_slate_home: Path, tmp_path: Path + ) -> None: + """Case A: with valid agentcore config the per-prompt hook reaches + BOTH fake memories on the wire (EPI reflections fan-out via + backend.retrieve + SEM via backend.semantic_list) and injects a + block into the envelope.""" + bm_home = _bm_home(clean_slate_home) + with FakeAgentCore() as fake: + write_fake_agentcore_json(bm_home) + fake.set_response( + "ListMemoryRecords", + {"memoryRecordSummaries": [_DOCKER_REFLECTION_SUMMARY]}, + ) + env = agentcore_env( + clean_slate_home, + fake.port, + BETTER_MEMORY_CONTEXT_INJECT_MODE="userprompt", + ) + + rc, out, err = run_hook( + "better_memory.hooks.contextual_inject", + _inject_payload(tmp_path), + env, + ) + assert rc == 0 + assert "Traceback" not in err + envelope = json.loads(out) + hook_output = envelope["hookSpecificOutput"] + assert hook_output["hookEventName"] == "UserPromptSubmit" + rendered = hook_output["additionalContext"] + assert " None: + """Case B — KNOWN-DEFECT PIN (design section 4 item 6): ID vars + unset (exactly the state a user who followed the documented setup + is in) on a clean slate → every prompt silently gets the empty + envelope, zero wire traffic, AND record_hook_error's connect() + leaves a stray schema-less memory.db behind while the hook_errors + INSERT silently no-ops (no table). Flips when the idvar gate is + fixed or record_hook_error stops creating the stray DB.""" + bm_home = _bm_home(clean_slate_home) + with FakeAgentCore() as fake: + env = agentcore_env( + clean_slate_home, + fake.port, + BETTER_MEMORY_CONTEXT_INJECT_MODE="userprompt", + **remove_dummy_id_vars_pins(), + ) + + rc, out, err = run_hook( + "better_memory.hooks.contextual_inject", + _inject_payload(tmp_path), + env, + ) + assert rc == 0 + assert err == "" + assert json.loads(out) == { + "hookSpecificOutput": { + "hookEventName": "UserPromptSubmit", + "additionalContext": "", + } + } + assert fake.requests == [] + + # Stray schema-less DB is the ONLY artifact; no state/ litter. + assert {child.name for child in bm_home.iterdir()} == {"memory.db"} + assert _table_names(bm_home / "memory.db") == set() + + def test_case_c_misconfig_premigrated_db_records_one_hook_error_row( + self, clean_slate_home: Path, tmp_path: Path + ) -> None: + """Case C: same misconfig against a pre-migrated memory.db → + exactly one hook_errors row (hook_name='contextual_inject', + exception_type='ValueError', exception_message naming both ID + vars — column name per migration 0005_phase_c.sql); state/ still + never created (get_config raises before the SeenStore block).""" + from better_memory.db.connection import connect + from better_memory.db.schema import apply_migrations + + bm_home = _bm_home(clean_slate_home) + conn = connect(bm_home / "memory.db") + try: + apply_migrations(conn) + finally: + conn.close() + + with FakeAgentCore() as fake: + env = agentcore_env( + clean_slate_home, + fake.port, + BETTER_MEMORY_CONTEXT_INJECT_MODE="userprompt", + **remove_dummy_id_vars_pins(), + ) + rc, out, err = run_hook( + "better_memory.hooks.contextual_inject", + _inject_payload(tmp_path), + env, + ) + assert rc == 0 + assert err == "" + assert json.loads(out) == { + "hookSpecificOutput": { + "hookEventName": "UserPromptSubmit", + "additionalContext": "", + } + } + assert fake.requests == [] + + assert not (bm_home / "state").exists() + with closing(sqlite3.connect(bm_home / "memory.db")) as check: + rows = check.execute( + "SELECT hook_name, exception_type, exception_message " + "FROM hook_errors" + ).fetchall() + assert len(rows) == 1 + hook_name, exception_type, exception_message = rows[0] + assert hook_name == "contextual_inject" + assert exception_type == "ValueError" + # Both var names (taken from the single helper's source of truth — + # never spelled out here; the tripwire grep-pins the literals). + for var_name in DUMMY_ID_VARS: + assert var_name in exception_message + + +# --------------------------------------------------------------------------- +# D7 — region split-brain (KNOWN-DEFECT PIN) +# --------------------------------------------------------------------------- + + +class TestRegionSplitBrainPin: + async def test_server_signs_env_default_while_hook_signs_json_region( + self, clean_slate_home: Path, tmp_path: Path + ) -> None: + """KNOWN-DEFECT PIN (design section 4 item 4): with agentcore.json + region=us-east-1 and BETTER_MEMORY_AGENTCORE_REGION unset, the two + planes disagree — (a) the MCP server's wired ``memory.retrieve`` + signs SigV4 with eu-west-2 (the env DEFAULT from + better_memory/config.py) while consuming the json's memory id + (MEM-EPI-JSON on the wire; the dummy env value never is); + (b) the session_close hook signs us-east-1 (json-derived). + A single-source-of-truth fix flips both halves visibly.""" + bm_home = _bm_home(clean_slate_home) + with FakeAgentCore() as fake: + write_fake_agentcore_json( + bm_home, + region="us-east-1", + episodic_memory_id="MEM-EPI-JSON", + semantic_memory_id="MEM-SEM-JSON", + ) + fake.set_response("ListMemoryRecords", {"memoryRecordSummaries": []}) + # The split-brain condition: env region var ABSENT. + env = agentcore_env( + clean_slate_home, + fake.port, + BETTER_MEMORY_AGENTCORE_REGION=None, + ) + + # (a) server plane — memory.retrieve is the WIRED trigger. + async with mcp_session(env) as session: + result = await session.call_tool("memory.retrieve", {}) + assert not result.isError + + server_requests = fake.requests_for("ListMemoryRecords") + assert len(server_requests) == 3 + for request in server_requests: + assert request.sigv4_region == "eu-west-2" # env default + assert request.sigv4_region != "us-east-1" # NOT the json's + # Runtime IDs come from agentcore.json — presence of the + # real id, plus absence of the dummy env value, proves the + # dead env vars are never consumed. + assert "MEM-EPI-JSON" in request.path + assert DUMMY_EPISODIC_MEMORY_ID not in request.text() + + # (b) hook plane — session_close signs with the json's region. + fake.clear() + rc, out, _err = run_hook( + "better_memory.hooks.session_close", + { + "session_id": "e2e-session-1", + "cwd": str(tmp_path / "proj"), + "hook_event_name": "Stop", + }, + env, + ) + assert rc == 0 + assert out == "" + closures = fake.requests_for("CreateEvent") + assert len(closures) == 1 + assert closures[0].sigv4_region == "us-east-1" + assert "MEM-EPI-JSON" in closures[0].path diff --git a/tests/e2e/test_hooks_contracts.py b/tests/e2e/test_hooks_contracts.py new file mode 100644 index 0000000..3577a5e --- /dev/null +++ b/tests/e2e/test_hooks_contracts.py @@ -0,0 +1,212 @@ +"""E2E contracts for the synchronous per-prompt hook (design task 5). + +Scenario ``e2e-sqlite-contextual-inject-contract`` from +``docs/superpowers/specs/2026-07-12-e2e-clean-slate-smoke-design.md`` §1C: + +(a) virgin home + **default mode** (env var unset = ``'both'``, + ``config.py _DEFAULT_CONTEXT_INJECT_MODE``): exit 0, exactly one JSON + envelope line on stdout, never a traceback. A regression here breaks + *every prompt* in *every session* for every new user. +(b) migrated home + seeded matching semantic memory (seeded through the real + server via ``memory.semantic_observe``): injection envelope carries the + ```` block and a ``session_memory_exposure`` row lands + with ``source='contextual'``. +(c) ``mode=off``: exit 0, byte-exact empty envelope, ``BETTER_MEMORY_HOME`` + never created — verified zero side effects. + +Plus the outer never-fail guard branch (``except BaseException`` in +``better_memory/hooks/contextual_inject.py:main``) gets its own triggering +test — it is the mutation-smoke M3 sentinel surface (exactly-one-JSON-line +stdout). + +The degraded / heal SessionStart contracts (design C2/C3) are owned by the +sqlite-journey task and land in this module separately — do not fold them +into these classes. +""" + +from __future__ import annotations + +import json +import sqlite3 +from contextlib import closing +from pathlib import Path +from typing import Any + +from tests.e2e._env import isolated_env +from tests.e2e.conftest import mcp_session, run_hook + +INJECT_HOOK = "better_memory.hooks.contextual_inject" + +PROJECT_MEMORY_OPEN = '' +PROJECT_MEMORY_CLOSE = "" + + +def _prompt_payload(prompt: str, cwd: Path) -> dict[str, Any]: + """A UserPromptSubmit payload exactly as Claude Code delivers it.""" + return { + "hook_event_name": "UserPromptSubmit", + "prompt": prompt, + "session_id": "e2e-session-1", + "cwd": str(cwd), + } + + +def _single_envelope(stdout: str) -> dict[str, Any]: + """Enforce the hook's stdout contract: exactly one JSON line. + + This is the mutation-smoke M3 sentinel assertion (design §5): a + regression where the except path exits without printing the envelope + must flip this red, so the check is strict — one line, valid JSON, + ``hookSpecificOutput`` as the only top-level key. + """ + lines = stdout.strip().splitlines() + assert len(lines) == 1, f"expected exactly one JSON line on stdout, got: {stdout!r}" + parsed = json.loads(lines[0]) + assert isinstance(parsed, dict) + assert set(parsed) == {"hookSpecificOutput"} + return parsed + + +def _empty_envelope(event: str = "UserPromptSubmit") -> dict[str, Any]: + return {"hookSpecificOutput": {"hookEventName": event, "additionalContext": ""}} + + +class TestContextualInjectContract: + """e2e-sqlite-contextual-inject-contract cases (a)/(b)/(c).""" + + def test_a_default_mode_virgin_home_exits_zero_with_envelope( + self, clean_slate_home: Path + ) -> None: + """Case (a): a brand-new user's very first prompt, mode var unset. + + The default is 'both' (config.py), so the hook genuinely executes its + injection path against a home with no DB at all. Contract: exit 0, + single JSON envelope, no traceback. Nothing exists to inject, so + additionalContext is exactly ''. + """ + env = isolated_env(clean_slate_home) + # The default-mode leg is the point: the choke-point helper must not + # be pinning the mode var for us. + assert not any(k.upper() == "BETTER_MEMORY_CONTEXT_INJECT_MODE" for k in env) + + rc, out, err = run_hook( + INJECT_HOOK, + _prompt_payload("hello brand new better-memory world", clean_slate_home), + env, + ) + + assert rc == 0, err + assert "Traceback" not in err + assert _single_envelope(out) == _empty_envelope() + + def test_a_config_error_swallowed_envelope_still_printed( + self, clean_slate_home: Path + ) -> None: + """Guard branch: the outer never-fail wrapper in contextual_inject.main. + + An invalid BETTER_MEMORY_CONTEXT_INJECT_MODE value makes get_config() + raise ValueError *inside* the try — the hook must swallow it, + record_hook_error, and still print the envelope with exit 0. + + Anti-vacuity: record_hook_error's connect() creates a schema-less + memory.db on a virgin home (its INSERT then silently no-ops — same + stray-DB defect pinned at agentcore level in scenario D6). Asserting + that artifact proves the except path actually executed. + """ + env = isolated_env( + clean_slate_home, BETTER_MEMORY_CONTEXT_INJECT_MODE="aggressive" + ) + + rc, out, err = run_hook( + INJECT_HOOK, _prompt_payload("any prompt at all", clean_slate_home), env + ) + + assert rc == 0, err + assert "Traceback" not in err + assert _single_envelope(out) == _empty_envelope() + + # Proof the except path ran (not a silently-valid mode value): + db = clean_slate_home / ".better-memory" / "memory.db" + assert db.exists() + with closing(sqlite3.connect(db)) as conn: + (tables,) = conn.execute("SELECT COUNT(*) FROM sqlite_master").fetchone() + assert tables == 0 + + async def test_b_seeded_memory_injects_and_records_contextual_exposure( + self, clean_slate_home: Path, tmp_path: Path + ) -> None: + """Case (b): migrated home + one matching memory -> injection fires. + + Seeding recipe per design task 5: the memory goes in through the real + MCP server via memory.semantic_observe (server boot also migrates the + DB). The prompt shares 4 distinct whole-word keywords with the memory + content ('configure', 'widget', 'frobnicator', 'cache'), clearing the + default context_min_hits=2 floor in services/relevant.py. + """ + env = isolated_env(clean_slate_home) + errlog_path = tmp_path / "seed-server.stderr" # outside the fake home + content = ( + "Always configure the widget frobnicator cache with LRU eviction " + "in this project." + ) + with errlog_path.open("w", encoding="utf-8") as errlog: + async with mcp_session(env, errlog=errlog) as session: + result = await session.call_tool( + "memory.semantic_observe", {"content": content} + ) + assert not result.isError, result.content + memory_id = json.loads(result.content[0].text)["id"] + assert memory_id + + rc, out, err = run_hook( + INJECT_HOOK, + _prompt_payload( + "How do I configure the widget frobnicator cache?", clean_slate_home + ), + env, + ) + + assert rc == 0, err + envelope = _single_envelope(out) + assert envelope["hookSpecificOutput"]["hookEventName"] == "UserPromptSubmit" + ctx = envelope["hookSpecificOutput"]["additionalContext"] + assert PROJECT_MEMORY_OPEN in ctx + assert PROJECT_MEMORY_CLOSE in ctx + assert f"semantic {memory_id}" in ctx # the [kind id ...] meta tag + assert "widget frobnicator cache" in ctx # the rendered memory text + + # Exposure row wire shape per migration 0012 (session_memory_exposure + # recreated with source CHECK including 'contextual'). Subset column + # select — never the exact column set. + db = clean_slate_home / ".better-memory" / "memory.db" + with closing(sqlite3.connect(db)) as conn: + rows = conn.execute( + "SELECT session_id, memory_kind, memory_id " + "FROM session_memory_exposure WHERE source = 'contextual'" + ).fetchall() + assert ("e2e-session-1", "semantic", memory_id) in rows + + def test_c_mode_off_empty_envelope_and_zero_side_effects( + self, clean_slate_home: Path + ) -> None: + """Case (c): mode=off is a true no-op — exact empty envelope AND + BETTER_MEMORY_HOME never created (resolves phase-1 open question #8: + off-mode must not leave state/ or DB litter). The whole fake home + stays byte-empty. + """ + env = isolated_env( + clean_slate_home, BETTER_MEMORY_CONTEXT_INJECT_MODE="off" + ) + + rc, out, err = run_hook( + INJECT_HOOK, + _prompt_payload("a prompt that would otherwise match", clean_slate_home), + env, + ) + + assert rc == 0, err + assert "Traceback" not in err + assert _single_envelope(out) == _empty_envelope() + + assert not (clean_slate_home / ".better-memory").exists() + assert list(clean_slate_home.iterdir()) == [] diff --git a/tests/e2e/test_install_hooks.py b/tests/e2e/test_install_hooks.py new file mode 100644 index 0000000..731cc8e --- /dev/null +++ b/tests/e2e/test_install_hooks.py @@ -0,0 +1,712 @@ +"""E2E installer scenarios A1-A7 (design catalog section 1.A, T1). + +Subprocess-level tests of ``python -m better_memory.cli.install_hooks`` +against a truly clean-slate fake home built by ``isolated_env``. Unlike the +unit-level ``tests/cli/test_install_hooks.py`` (whose ``mock_home`` fixture +pre-creates ``.claude/`` and ``install-backups/``), these exercise the real +brand-new-user path: ``_atomic_write``'s parent mkdir, +``_resolve_user_skills_dir``'s mkdir, and the full argv -> argparse -> merge +-> atomic-write pipeline in a separate process. + +Scenario map (ids from ``2026-07-12-e2e-clean-slate-smoke-design.md``): + +* A1 ``e2e-install-fresh-clean-slate`` -> test_fresh_clean_slate_writes_exact_shapes +* A2 ``e2e-install-idempotent-rerun`` -> test_idempotent_rerun_byte_identical_no_dup +* A3 ``e2e-install-foreign-config-preserved`` -> test_foreign_config_preserved_and_legacy_scrubbed +* A4 ``e2e-install-malformed-claude-json-refused`` + -> test_malformed_claude_json_refuses_whole_install +* A5 ``e2e-install-backup-before-overwrite`` -> test_backup_before_overwrite +* A6 ``e2e-install-symlink-oserror-fallback`` -> test_symlink_oserror_fallback_warns_and_continues +* A7 ``e2e-install-symlink-replacement-ladder``-> test_symlink_replacement_ladder / _noop + +Interpreter paths deliberately contain an embedded space: the quoting in +``_hook_entry`` (``"{interpreter}" -m {module}``) is load-bearing for every +venv path with a space, and no other test in the repo passes one. +""" + +from __future__ import annotations + +import json +import os +import re +import subprocess +import sys +from dataclasses import dataclass +from pathlib import Path + +import pytest + +from better_memory.cli import install_hooks as installer_module +from tests.e2e._env import isolated_env + +# Embedded space is load-bearing: dropping the quotes in _hook_entry would +# pass any test using a space-free interpreter path. Values are never +# executed - the installer only serializes them into config. +VENV_PY = "C:/fake venv/python.exe" +VENV_PYW = "C:/fake venv/pythonw.exe" + +#: The exact source dir the installer symlinks from (mirrors +#: install_skill_symlinks' parents[2] resolution so the assertion cannot +#: drift from the implementation's own notion of "the repo"). +REPO_SKILLS_DIR = ( + Path(installer_module.__file__).resolve().parents[2] / ".claude" / "skills" +) + +SKILL_NAMES = ("better-memory-synthesize", "rate-session-memories") + +#: (event, module, expected async, expected group matcher) for all 5 managed +#: entries. Interpreter selection: needs_stdout hooks get VENV_PY, the rest +#: VENV_PYW (pythonw would silently null stdout and kill additionalContext). +EXPECTED_ENTRIES: dict[str, tuple[str, str, bool, str | None]] = { + "SessionStart": ("better_memory.hooks.session_bootstrap", VENV_PY, False, None), + "PostToolUse": ("better_memory.hooks.observer", VENV_PYW, True, "Write|Edit|Bash"), + "Stop": ("better_memory.hooks.session_close", VENV_PYW, True, None), + "UserPromptSubmit": ("better_memory.hooks.contextual_inject", VENV_PY, False, None), + "PreToolUse": ("better_memory.hooks.contextual_inject", VENV_PY, False, "Skill|Task|Write"), +} + + +@dataclass +class InstallHarness: + """One clean-slate fake home plus everything needed to run the installer.""" + + home: Path # the fake user home (empty at construction) + tmp: Path # pytest tmp_path - side artifacts live here + env: dict[str, str] # isolated_env-built child environment + home_arg: str # exact --home string passed (== BETTER_MEMORY_HOME) + + @property + def claude_json(self) -> Path: + return self.home / ".claude.json" + + @property + def settings_json(self) -> Path: + return self.home / ".claude" / "settings.json" + + @property + def skills_dir(self) -> Path: + return self.home / ".claude" / "skills" + + @property + def backups_dir(self) -> Path: + return self.home / ".better-memory" / "install-backups" + + def run(self) -> subprocess.CompletedProcess[str]: + """One real installer run: python -m better_memory.cli.install_hooks.""" + return _spawn( + self.env, + [sys.executable, "-m", "better_memory.cli.install_hooks", *self.installer_args], + ) + + @property + def installer_args(self) -> list[str]: + return [ + "--venv-py", VENV_PY, + "--venv-pyw", VENV_PYW, + "--home", self.home_arg, + ] + + +def _spawn( + env: dict[str, str], argv: list[str], *, timeout: float = 60 +) -> subprocess.CompletedProcess[str]: + """The module's single spawn site. ``env`` MUST come from isolated_env + (enforced by tests/e2e_meta/test_env_helper_contract.py contract C).""" + return subprocess.run( # noqa: S603 - test harness, fixed argv + argv, env=env, capture_output=True, text=True, timeout=timeout + ) + + +@pytest.fixture +def harness(clean_slate_home: Path, tmp_path: Path) -> InstallHarness: + home_arg = str(clean_slate_home / ".better-memory") + return InstallHarness( + home=clean_slate_home, + tmp=tmp_path, + env=isolated_env(clean_slate_home), + home_arg=home_arg, + ) + + +# --------------------------------------------------------------------- helpers + + +def bm_hook_entries(settings: dict) -> list[tuple[str, dict, dict]]: + """Every (event, group, entry) whose command references a bm hook module. + + Subset-tolerant by construction: foreign events/groups/entries are walked + but only better-memory module commands are collected. + """ + found: list[tuple[str, dict, dict]] = [] + for event, groups in settings.get("hooks", {}).items(): + for group in groups: + for entry in group.get("hooks", []): + if "better_memory.hooks." in entry.get("command", ""): + found.append((event, group, entry)) + return found + + +def assert_managed_shapes(settings: dict) -> None: + """The full 5-entry contract: distribution, quoting, py/pyw split, + async flags, and matchers.""" + entries = bm_hook_entries(settings) + assert len(entries) == 5, ( + f"expected exactly 5 better-memory hook entries, got {len(entries)}: " + f"{[(e, en['command']) for e, _, en in entries]}" + ) + by_event = {event: (group, entry) for event, group, entry in entries} + assert set(by_event) == set(EXPECTED_ENTRIES), ( + "expected exactly one better-memory entry per event in " + f"{sorted(EXPECTED_ENTRIES)}, got {sorted(e for e, _, _ in entries)}" + ) + for event, (module, interpreter, is_async, matcher) in EXPECTED_ENTRIES.items(): + group, entry = by_event[event] + assert entry["type"] == "command" + # Quoted interpreter with the embedded space intact, exact format. + assert entry["command"] == f'"{interpreter}" -m {module}', ( + f"{event}: bad command {entry['command']!r}" + ) + if is_async: + assert entry["async"] is True, f"{event}: async flag missing" + else: + assert "async" not in entry, f"{event}: unexpected async flag" + if matcher is None: + assert "matcher" not in group, f"{event}: unexpected matcher" + else: + assert group["matcher"] == matcher, f"{event}: bad matcher" + + +def assert_mcp_server_shape(claude_config: dict, harness: InstallHarness) -> None: + bm = claude_config["mcpServers"]["better-memory"] + assert bm["type"] == "stdio" + assert bm["command"] == VENV_PY + assert bm["args"] == ["-m", "better_memory.mcp"] + assert bm["env"]["BETTER_MEMORY_HOME"] == harness.home_arg + + +def can_symlink(base: Path) -> bool: + """Runtime capability probe: True iff this process may create dir + symlinks (POSIX; Windows with Developer Mode or elevation).""" + probe_target = base / "symlink-probe-target" + probe_target.mkdir(exist_ok=True) + probe_link = base / "symlink-probe-link" + try: + probe_link.symlink_to(probe_target, target_is_directory=True) + except OSError: + return False + probe_link.unlink() + return True + + +# ------------------------------------------------------ A1: fresh clean slate + + +def test_fresh_clean_slate_writes_exact_shapes(harness: InstallHarness) -> None: + """A1 ``e2e-install-fresh-clean-slate``. + + Truly clean slate (no .claude/, no .claude.json, no .better-memory): + one subprocess run writes both targets with the exact managed shapes and + zero side writes. Traps: needs_stdout interpreter swap (pythonw silently + kills additionalContext), dropped interpreter quoting, missing + UserPromptSubmit/PreToolUse merge branches, the installer growing a + DB-creating side effect. + """ + # Preconditions: nothing pre-created. + assert list(harness.home.iterdir()) == [] + symlinks_expected = can_symlink(harness.tmp) + + proc = harness.run() + + assert proc.returncode == 0, proc.stderr + assert proc.stdout.count("(created fresh)") == 2, proc.stdout + assert "[install_hooks] Restart Claude Code" in proc.stdout + + # .claude.json subset (foreign keys tolerated - none here, but the + # assertion never enumerates the full key set). + claude_config = json.loads(harness.claude_json.read_text(encoding="utf-8")) + assert_mcp_server_shape(claude_config, harness) + + # settings.json: the full 5-entry contract. + settings = json.loads(harness.settings_json.read_text(encoding="utf-8")) + assert_managed_shapes(settings) + + # Zero DB / runtime side writes: the installer must never touch storage. + assert list(harness.home.rglob("memory.db")) == [] + assert list(harness.home.rglob("knowledge.db")) == [] + # Nothing to back up on a clean slate -> _backup no-ops, dir not created. + assert not harness.backups_dir.exists() + # No _atomic_write litter next to either target. + assert list(harness.home.rglob("*.tmp")) == [] + + # Skill symlinks, capability-probed on both branches. + if symlinks_expected: + for name in SKILL_NAMES: + link = harness.skills_dir / name + assert link.is_symlink(), f"{name} not symlinked" + assert link.resolve() == (REPO_SKILLS_DIR / name).resolve() + assert "WARN skill symlink skipped" not in proc.stderr + else: + # Locked-down host: the documented degraded path must have fired. + assert "WARN skill symlink skipped" in proc.stderr + + +# --------------------------------------------------- A2: idempotent re-runs + + +def test_idempotent_rerun_byte_identical_no_dup(harness: InstallHarness) -> None: + """A2 ``e2e-install-idempotent-rerun``. + + Runs 2 and 3 are byte-identical for BOTH targets; the managed entry + count stays exactly 5. Traps: REMOVE-pass predicate drift (exact-match + vs the module-path substring at install_hooks.py:167) appending duplicate + hook groups per re-run; serialization drift (sort_keys/indent changes). + """ + proc1 = harness.run() + assert proc1.returncode == 0, proc1.stderr + claude_1 = harness.claude_json.read_bytes() + settings_1 = harness.settings_json.read_bytes() + + proc2 = harness.run() + assert proc2.returncode == 0, proc2.stderr + claude_2 = harness.claude_json.read_bytes() + settings_2 = harness.settings_json.read_bytes() + + proc3 = harness.run() + assert proc3.returncode == 0, proc3.stderr + claude_3 = harness.claude_json.read_bytes() + settings_3 = harness.settings_json.read_bytes() + + # Byte identity across runs 1-3 for both targets. Three runs catch + # once-stable-then-drifting bugs a single re-run misses. + assert settings_2 == settings_1 + assert settings_3 == settings_2 + assert claude_2 == claude_1 + assert claude_3 == claude_2 + + # Still exactly 5 managed entries - no duplicate groups accumulated. + settings = json.loads(settings_3.decode("utf-8")) + assert len(bm_hook_entries(settings)) == 5 + + # Re-runs back up the previous run's output instead of "(created fresh)". + assert "(created fresh)" not in proc2.stdout + assert proc2.stdout.count("(backup:") == 2, proc2.stdout + + # At least one settings backup snapshots a state that already contained + # the 5 managed entries (run 1's output). "At least one", never exact + # counts: the 1-second timestamp granularity makes same-second re-runs + # overwrite the same .bak name. + settings_baks = list(harness.backups_dir.glob("settings.json.*.bak")) + assert settings_baks, "no settings.json backups written on re-run" + assert any( + len(bm_hook_entries(json.loads(bak.read_text(encoding="utf-8")))) == 5 + for bak in settings_baks + ) + + +# ------------------------------- A3: foreign config preserved + legacy scrub + + +def test_foreign_config_preserved_and_legacy_scrubbed( + harness: InstallHarness, +) -> None: + """A3 ``e2e-install-foreign-config-preserved``. + + Seeded foreign servers/hooks/top-level keys and the user's custom + BETTER_MEMORY_HOME survive; the command path is refreshed; legacy + ``session_start``/``session_retrieve`` entries are scrubbed while + co-resident user hooks in the same group survive (legacy seeds folded in + per the judge round - no standalone legacy scenario). Traps: + ``env.setdefault`` -> unconditional assignment (repoints the user's data + dir), dict rebuild dropping ``model``/foreign events, + ``_LEGACY_HOOK_MODULES`` deletion, group-level (vs entry-level) removal. + """ + seeded_other_server = {"type": "stdio", "command": "/x"} + seeded_top_level = {"a": 1} + harness.claude_json.write_text( + json.dumps( + { + "mcpServers": { + "other-server": seeded_other_server, + "better-memory": { + "type": "stdio", + "command": "/stale/old-python", + "args": ["-m", "better_memory.mcp"], + "env": { + "BETTER_MEMORY_HOME": "D:/custom/bm-home", + "EXTRA_VAR": "keep-me", + }, + }, + }, + "someOtherTopLevel": seeded_top_level, + } + ), + encoding="utf-8", + ) + seeded_precompact_group = { + "hooks": [{"type": "command", "command": "echo foreign-event"}], + } + harness.settings_json.parent.mkdir(parents=True) + harness.settings_json.write_text( + json.dumps( + { + "model": "opus", + "hooks": { + # Legacy bm entry co-resident with a user hook: the + # REMOVE pass must strip per-entry, not per-group. + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": '"/old/python" -m better_memory.hooks.session_start', + }, + {"type": "command", "command": "echo user-owned-hook"}, + ], + }, + ], + # Legacy-only group: emptied by removal -> dropped. + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": ( + '"/old/python" -m ' + "better_memory.hooks.session_retrieve" + ), + }, + ], + }, + ], + # Stale current-module entry: refreshed via remove+add. + "PostToolUse": [ + { + "matcher": "Write|Edit|Bash", + "hooks": [ + { + "type": "command", + "command": '"/older/pythonw" -m better_memory.hooks.observer', + }, + ], + }, + ], + # Foreign event the installer knows nothing about. + "PreCompact": [seeded_precompact_group], + }, + } + ), + encoding="utf-8", + ) + + proc = harness.run() + assert proc.returncode == 0, proc.stderr + + claude_config = json.loads(harness.claude_json.read_text(encoding="utf-8")) + # Foreign server + foreign top-level key deep-equal their seeds. + assert claude_config["mcpServers"]["other-server"] == seeded_other_server + assert claude_config["someOtherTopLevel"] == seeded_top_level + bm = claude_config["mcpServers"]["better-memory"] + # User's custom home WINS over --home (setdefault semantics)... + assert bm["env"]["BETTER_MEMORY_HOME"] == "D:/custom/bm-home" + assert bm["env"]["EXTRA_VAR"] == "keep-me" + # ...while the command path is refreshed from the stale seed. + assert bm["command"] == VENV_PY + + settings = json.loads(harness.settings_json.read_text(encoding="utf-8")) + # Top-level foreign key + foreign event survive dict(existing). + assert settings["model"] == "opus" + assert settings["hooks"]["PreCompact"] == [seeded_precompact_group] + + # Legacy scrub: no command anywhere references a legacy module. (Safe + # against canonical entries: 'session_start' is not a substring of + # 'session_bootstrap'.) + all_commands = [ + entry.get("command", "") + for groups in settings["hooks"].values() + for group in groups + for entry in group.get("hooks", []) + ] + assert not any("better_memory.hooks.session_start" in c for c in all_commands) + assert not any("better_memory.hooks.session_retrieve" in c for c in all_commands) + + # Co-resident user hook survived the per-entry strip; its group no + # longer carries the legacy entry; the canonical group was appended + # alongside -> exactly 2 SessionStart groups. + session_start_groups = settings["hooks"]["SessionStart"] + assert len(session_start_groups) == 2, session_start_groups + user_groups = [ + g + for g in session_start_groups + if any(e["command"] == "echo user-owned-hook" for e in g["hooks"]) + ] + assert len(user_groups) == 1 + assert user_groups[0]["hooks"] == [ + {"type": "command", "command": "echo user-owned-hook"} + ] + + # Legacy-only UserPromptSubmit group dropped; exactly the canonical one. + upsubmit_groups = settings["hooks"]["UserPromptSubmit"] + assert len(upsubmit_groups) == 1, upsubmit_groups + + # Stale observer refreshed: exactly one observer entry, current VENV_PYW. + observer_commands = [c for c in all_commands if "better_memory.hooks.observer" in c] + assert observer_commands == [f'"{VENV_PYW}" -m better_memory.hooks.observer'] + assert not any("/older/pythonw" in c for c in all_commands) + + # Full canonical shape holds on top of the preserved foreign config. + assert_managed_shapes(settings) + + +# --------------------------------------- A4: malformed .claude.json refused + + +def test_malformed_claude_json_refuses_whole_install( + harness: InstallHarness, +) -> None: + """A4 ``e2e-install-malformed-claude-json-refused``. + + Malformed ``~/.claude.json`` (the FIRST target - the direction existing + unit tests don't cover; they seed malformed settings.json): exit 1, + path+lineno+remediation on stderr, BOTH files byte-unchanged, no + backups, no ``*.tmp`` litter. Pins the swallow-and-treat-as-``{}`` + hazard: a catch-JSONDecodeError-continue-with-``{}`` change would + atomically replace the user's entire MCP config. + """ + malformed = b'{"mcpServers": {broken' + harness.claude_json.write_bytes(malformed) + harness.settings_json.parent.mkdir(parents=True) + valid_settings = b'{"hooks": {}}' + harness.settings_json.write_bytes(valid_settings) + + proc = harness.run() + + assert proc.returncode == 1 + # Actionable remediation: :: + fix-then-re-run. + assert re.search( + rf"{re.escape(str(harness.claude_json))}:\d+:", proc.stderr + ), proc.stderr + assert "Fix the file then re-run" in proc.stderr + + # The malformed file was NOT replaced with a minimal managed config. + assert harness.claude_json.read_bytes() == malformed + # The VALID second target was not merged either: validation of ALL + # targets precedes ANY write (install_hooks.py main() two-pass loop). + assert harness.settings_json.read_bytes() == valid_settings + + # No backup taken on a refused install; no atomic-write litter. + assert not harness.backups_dir.exists() + assert list(harness.home.rglob("*.tmp")) == [] + + # Deliberate NON-assertion: /.claude/skills MAY exist - + # install_skill_symlinks() runs before validation. Pinning only the two + # JSON targets keeps this honest on symlink-capable and locked-down + # hosts alike. + + +# ------------------------------------------- A5: backup before overwrite + + +def test_backup_before_overwrite(harness: InstallHarness) -> None: + """A5 ``e2e-install-backup-before-overwrite``. + + Backups land in ``$BETTER_MEMORY_HOME/install-backups/`` with + timestamped names containing the PRE-run bytes; the dir is auto-created. + Trap: moving the ``_backup`` call after ``_atomic_write`` (backup-of- + result destroys the user's only rollback artifact), and drift in the + backup location/name format that docs point users at. + """ + claude_seed = b'{"mcpServers": {"sentinel": {"command": "/s"}}}' + settings_seed = ( + b'{"hooks": {"Stop": [{"hooks": ' + b'[{"type": "command", "command": "echo sentinel-stop"}]}]}}' + ) + harness.claude_json.write_bytes(claude_seed) + harness.settings_json.parent.mkdir(parents=True) + harness.settings_json.write_bytes(settings_seed) + assert not harness.backups_dir.exists() # precondition: auto-creation + + proc = harness.run() + + assert proc.returncode == 0, proc.stderr + # Relative-path rendering; the prefix is separator-agnostic (the sep + # comes after the matched substring on both OSes). + assert proc.stdout.count("(backup: install-backups") == 2, proc.stdout + + # install-backups auto-created (mkdir parents in _backup); exactly 2 + # backups - one per pre-existing target, single run so the 1-second + # timestamp cannot collide. + assert harness.backups_dir.is_dir() + backups = sorted(p.name for p in harness.backups_dir.iterdir()) + assert len(backups) == 2, backups + claude_baks = [n for n in backups if re.fullmatch(r"\.claude\.json\.\d{8}-\d{6}\.bak", n)] + settings_baks = [n for n in backups if re.fullmatch(r"settings\.json\.\d{8}-\d{6}\.bak", n)] + assert len(claude_baks) == 1 and len(settings_baks) == 1, backups + + # Backups contain the PRE-run bytes exactly - captured before merge, + # never the merged output. + assert (harness.backups_dir / claude_baks[0]).read_bytes() == claude_seed + assert (harness.backups_dir / settings_baks[0]).read_bytes() == settings_seed + + # And the live files now hold sentinel content AND managed entries - + # proving backup+merge+write ordering, not backup-of-result. + claude_config = json.loads(harness.claude_json.read_text(encoding="utf-8")) + assert claude_config["mcpServers"]["sentinel"] == {"command": "/s"} + assert_mcp_server_shape(claude_config, harness) + settings = json.loads(harness.settings_json.read_text(encoding="utf-8")) + stop_commands = [ + e["command"] for g in settings["hooks"]["Stop"] for e in g["hooks"] + ] + assert "echo sentinel-stop" in stop_commands + assert_managed_shapes(settings) + + +# ------------------------------------- A6: symlink OSError(1314) fallback + + +_SYMLINK_DENIED_DRIVER = """\ +import pathlib +import sys + + +def _deny(self, *args, **kwargs): + raise OSError(1314, "A required privilege is not held by the client") + + +# Patch BEFORE importing the installer: link.symlink_to resolves through the +# class, so every skill's symlink attempt raises the Windows no-Developer- +# Mode error deterministically on every host (incl. POSIX and Dev-Mode +# Windows, where a real symlink would succeed and skip the branch). +pathlib.Path.symlink_to = _deny + +from better_memory.cli.install_hooks import main + +main(sys.argv[1:]) +""" + + +def test_symlink_oserror_fallback_warns_and_continues( + harness: InstallHarness, +) -> None: + """A6 ``e2e-install-symlink-oserror-fallback``. + + Driver script patches ``Path.symlink_to`` to raise ``OSError(1314)`` + (Windows-without-Developer-Mode simulation): exit 0, one WARN per skill + plus the Developer Mode remediation on stderr, both configs still fully + written. Trap: narrowing/removing the ``except OSError`` - symlinks run + BEFORE the JSON writes, so an unhandled raise is a total install failure + for exactly the locked-down-Windows population the fallback serves. + """ + driver = harness.tmp / "symlink_denied_driver.py" # outside the fake home + driver.write_text(_SYMLINK_DENIED_DRIVER, encoding="utf-8") + + proc = _spawn( + harness.env, + [sys.executable, str(driver), *harness.installer_args], + ) + + # Symlink failure is non-fatal: the core install completes. + assert proc.returncode == 0, proc.stderr + assert proc.stderr.count("WARN skill symlink skipped") >= 2, proc.stderr + assert "Developer Mode" in proc.stderr + assert "[install_hooks] Restart Claude Code" in proc.stdout + + # Degraded skills, intact core: both configs carry the full shapes. + claude_config = json.loads(harness.claude_json.read_text(encoding="utf-8")) + assert_mcp_server_shape(claude_config, harness) + settings = json.loads(harness.settings_json.read_text(encoding="utf-8")) + assert_managed_shapes(settings) + + # Dir creation succeeded (separate from link creation), links did not. + assert harness.skills_dir.is_dir() + for name in SKILL_NAMES: + assert not (harness.skills_dir / name).exists() + + +# --------------------------------------- A7: symlink replacement ladder + + +LADDER_SKILL = "better-memory-synthesize" + + +def _seed_obstacle(kind: str, link: Path, tmp: Path) -> None: + link.parent.mkdir(parents=True, exist_ok=True) + if kind == "wrong-symlink": + wrong_target = tmp / "wrong-target" + wrong_target.mkdir() + link.symlink_to(wrong_target, target_is_directory=True) + elif kind == "plain-file": + link.write_text("user-materialized skill file", encoding="utf-8") + elif kind == "real-dir-with-sentinel": + link.mkdir(parents=True) + (link / "SENTINEL.txt").write_text("user sentinel content", encoding="utf-8") + else: # pragma: no cover - parametrize guards this + raise AssertionError(kind) + + +@pytest.mark.parametrize( + "obstacle", ["wrong-symlink", "plain-file", "real-dir-with-sentinel"] +) +def test_symlink_replacement_ladder( + harness: InstallHarness, obstacle: str +) -> None: + """A7 ``e2e-install-symlink-replacement-ladder`` (gap, per judges). + + Capability-probed: a pre-existing wrong symlink / plain file / real dir + with a sentinel at the skill path each get replaced by a correct symlink + with NO warn; the sentinel is gone (pins the documented no-backup + destructive contract, so any future softening is a deliberate change). + Trap: removing the pre-clearing makes ``symlink_to`` raise + ``FileExistsError`` - an OSError - silently converting every upgrade + into perpetual "WARN skipped": skills never update again. + """ + if not can_symlink(harness.tmp): + pytest.skip("symlink capability unavailable (Windows without Developer Mode)") + + link = harness.skills_dir / LADDER_SKILL + _seed_obstacle(obstacle, link, harness.tmp) + + proc = harness.run() + + assert proc.returncode == 0, proc.stderr + assert "WARN skill symlink skipped" not in proc.stderr + + expected_source = (REPO_SKILLS_DIR / LADDER_SKILL).resolve() + assert link.is_symlink(), f"{obstacle}: obstacle not replaced by symlink" + assert link.resolve() == expected_source + + if obstacle == "real-dir-with-sentinel": + # The seeded dir was rmtree'd without backup: the sentinel resolves + # through the new symlink into the repo skill dir, where it must + # not exist - and must never have been copied into the repo. + assert not (link / "SENTINEL.txt").exists() + assert not (expected_source / "SENTINEL.txt").exists() + + +def test_symlink_already_correct_is_noop(harness: InstallHarness) -> None: + """A7 (continued): an already-correct link is a no-op (lstat compare). + + Trap: an ADD-style rewrite that unconditionally unlinks+recreates would + churn the link on every install; the lstat identity assertion flips. + """ + if not can_symlink(harness.tmp): + pytest.skip("symlink capability unavailable (Windows without Developer Mode)") + + link = harness.skills_dir / LADDER_SKILL + link.parent.mkdir(parents=True) + link.symlink_to(REPO_SKILLS_DIR / LADDER_SKILL, target_is_directory=True) + # ino + ctime identify the link object itself; a delete/recreate cannot + # reproduce either (NTFS/ext ctime resolution is far below the >100ms + # subprocess runtime). + before = os.lstat(link) + before_identity = (before.st_ino, before.st_ctime_ns) + + proc = harness.run() + + assert proc.returncode == 0, proc.stderr + assert "WARN skill symlink skipped" not in proc.stderr + assert link.is_symlink() + assert link.resolve() == (REPO_SKILLS_DIR / LADDER_SKILL).resolve() + after = os.lstat(link) + assert (after.st_ino, after.st_ctime_ns) == before_identity, ( + "already-correct symlink was recreated instead of no-op'd" + ) diff --git a/tests/e2e/test_setup_sh.py b/tests/e2e/test_setup_sh.py new file mode 100644 index 0000000..57510be --- /dev/null +++ b/tests/e2e/test_setup_sh.py @@ -0,0 +1,596 @@ +"""E2E scenarios B1-B4: ``scripts/setup.sh`` (design section 1.B). + +Scenarios (2026-07-12-e2e-clean-slate-smoke-design.md, catalog 1.B): + +* **B1** ``e2e-setup-headless-decline-completes`` — headless run to + completion: layout dirs, install_hooks invoked, win_path-correct command + in ``.claude.json``, py/pyw split survives the bash->argparse round-trip, + progress markers in order. +* **B2** ``e2e-setup-eof-aborts-all-or-nothing`` — closed stdin dies at the + Ollama ``read -rp`` under ``set -euo pipefail`` BEFORE layout and + install_hooks (product bug #11 pinned as all-or-nothing). Self-skips on + hosts where ollama is detectable (this dev machine); runs on ollama-free + CI. +* **B3** ``e2e-setup-default-home-derivation`` — ``BETTER_MEMORY_HOME`` + absent: the ``${VAR:-$HOME/.better-memory}`` default is derived and + propagated via ``--home`` into the ``mcpServers`` env block and the + install-backups location. +* **B4** ``e2e-setup-install-hooks-failure-propagates`` — malformed seeded + ``.claude.json``: exit 1, remediation text, no final ``Done.``, layout + stage completed (stage-order pin), target files untouched. + +Containment (design harness gap-1, embedded here): every run redirects +``UV_PROJECT_ENVIRONMENT`` to a module-shared tmp venv and pins +``UV_CACHE_DIR``/``UV_PYTHON_INSTALL_DIR`` to the *host's* warm locations +(isolated_env strips LOCALAPPDATA/APPDATA, which uv derives them from — an +unpinned run would use a cold cache under the fake home and hit the +network). The real repo ``.venv/pyvenv.cfg`` is hashed before/after every +run. The first run in the module bears one real ``uv sync`` into the tmp +venv (warm-cache); later runs are no-ops against the same venv. + +Network shims: ``curl`` (always exits 1) and ``ollama`` (no-op exit 0) are +**exported bash functions** injected by a driver script, not PATH files — +Git Bash prepends ``/mingw64/bin``/``/usr/bin`` to any inherited PATH, so a +PATH shim can never shadow the real MSYS curl (verified empirically). +Functions shadow everything. The failing curl forces the daemon-unreachable +warn-and-continue branch (setup.sh:165-172) on every host, so +``ollama pull`` can never run — no network, no dependence on host daemon +state. + +skipif: bash absent (contract test D in tests/e2e_meta checks for the +``shutil.which("bash")`` probe) or uv absent. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +import shutil +import subprocess +import sys +from dataclasses import dataclass +from pathlib import Path + +import pytest + +from tests.e2e._env import isolated_env + +REPO = Path(__file__).resolve().parents[2] +SETUP_SH = REPO / "scripts" / "setup.sh" +_WIN = sys.platform == "win32" + +#: setup.sh:107-111 — the interpreters whose (win_path-converted) paths the +#: installer writes into the config files. +VENV_PY = REPO / ".venv" / ("Scripts/python.exe" if _WIN else "bin/python") +VENV_PYW = REPO / ".venv" / ("Scripts/pythonw.exe" if _WIN else "bin/python") + +_ANSI = re.compile(r"\x1b\[[0-9;]*m") + + +def _find_bash() -> str | None: + """Locate a Git-Bash-flavored bash. + + On Windows, ``shutil.which("bash")`` is a trap twice over: it can find + WSL's ``System32\\bash.exe`` (a different OS) or stray shims (this dev + machine has a ``bash.exe`` inside a Python Scripts dir). Prefer paths + derived from the git installation, fall back to which() minus System32. + """ + if sys.platform != "win32": + return shutil.which("bash") + candidates: list[Path] = [] + git = shutil.which("git") + if git: + git_root = Path(git).resolve().parent.parent # /cmd/git.exe -> + candidates += [git_root / "bin" / "bash.exe", git_root / "usr" / "bin" / "bash.exe"] + for var in ("ProgramFiles", "ProgramW6432", "ProgramFiles(x86)"): + root = os.environ.get(var) + if root: + candidates.append(Path(root) / "Git" / "bin" / "bash.exe") + local = os.environ.get("LOCALAPPDATA") + if local: + candidates.append(Path(local) / "Programs" / "Git" / "bin" / "bash.exe") + for cand in candidates: + if cand.is_file(): + return str(cand) + which = shutil.which("bash") + if which and "system32" not in which.lower(): + return which + return None + + +BASH = _find_bash() + +pytestmark = [ + pytest.mark.skipif(BASH is None, reason="bash (Git Bash / POSIX) not available"), + pytest.mark.skipif(shutil.which("uv") is None, reason="uv not on PATH"), +] + + +def _posix(p: Path) -> str: + """Forward-slash form — safe to hand to bash as a script argument.""" + return str(p).replace("\\", "/") + + +def _host_uv_dirs() -> dict[str, str]: + """The host's real (warm) uv cache + managed-python dirs. + + Computed from the outer environment instead of spawning ``uv cache dir`` + (spawns must carry an isolated env, which would strip the very variables + the answer depends on). Mirrors uv's documented resolution: explicit env + var, else LOCALAPPDATA/APPDATA on Windows, XDG dirs elsewhere. + """ + cache = os.environ.get("UV_CACHE_DIR") + pydir = os.environ.get("UV_PYTHON_INSTALL_DIR") + if not cache: + if _WIN: + local = os.environ.get("LOCALAPPDATA", str(Path.home() / "AppData" / "Local")) + cache = str(Path(local) / "uv" / "cache") + else: + xdg = os.environ.get("XDG_CACHE_HOME", str(Path.home() / ".cache")) + cache = str(Path(xdg) / "uv") + if not pydir: + if _WIN: + roaming = os.environ.get("APPDATA", str(Path.home() / "AppData" / "Roaming")) + pydir = str(Path(roaming) / "uv" / "python") + else: + xdg_data = os.environ.get("XDG_DATA_HOME", str(Path.home() / ".local" / "share")) + pydir = str(Path(xdg_data) / "uv" / "python") + return {"UV_CACHE_DIR": cache, "UV_PYTHON_INSTALL_DIR": pydir} + + +def _base_python() -> str: + """A concrete non-shim CPython — the base interpreter behind pytest's venv. + + setup.sh's ``python``/``python3`` version probe and uv's interpreter + discovery must never execute PATH *shims*: on Windows hosts with the + Python 3.14 install manager, invoking its shim under an env with + LOCALAPPDATA stripped makes it auto-install a full CPython runtime into + ``/Python`` and Start Menu links under the fake profile (observed + empirically — potentially a network download per run). + """ + base = Path(sys.base_prefix) + candidates = ( + [base / "python.exe"] if _WIN else [base / "bin" / "python3", base / "bin" / "python"] + ) + for cand in candidates: + if cand.is_file(): + return str(cand) + return sys.executable + + +def _pyvenv_state() -> bytes | None: + """Fingerprint of the real repo venv (None when absent, e.g. bare CI).""" + cfg = REPO / ".venv" / "pyvenv.cfg" + if not cfg.exists(): + return None + return hashlib.sha256(cfg.read_bytes()).digest() + + +def _clean(text: str) -> str: + return _ANSI.sub("", text) + + +@dataclass(frozen=True) +class SetupRun: + rc: int + stdout: str + stderr: str + home: Path + cwd: Path + bash_version: str + + @property + def combined(self) -> str: + return self.stdout + "\n" + self.stderr + + def lines(self) -> list[str]: + return _clean(self.combined).splitlines() + + +def _run_setup( + env: dict[str, str], + *, + home: Path, + work: Path, + stdin: str | None, + ollama_shim: bool, + timeout: float = 600, +) -> SetupRun: + """Run scripts/setup.sh through the shim driver and return the outcome. + + ``stdin=None`` means closed stdin (EOF at the first ``read``); a string + is piped verbatim. ``ollama_shim=False`` leaves ``ollama`` undefined so + the interactive prompt branch (setup.sh:130-155) is reachable. + + Containment is asserted around every run: the real repo + ``.venv/pyvenv.cfg`` must be byte-identical afterwards (setup.sh runs + ``uv sync`` against the repo — UV_PROJECT_ENVIRONMENT must have + redirected it). + """ + pybin = _posix(Path(_base_python())) + shim_lines = [ + "curl() { return 1; }", + f'python() {{ "{pybin}" "$@"; }}', + f'python3() {{ "{pybin}" "$@"; }}', + "export -f curl python python3", + ] + if ollama_shim: + shim_lines += ["ollama() { return 0; }", "export -f ollama"] + driver = work / "driver.sh" + driver.write_text( + "\n".join( + [ + *shim_lines, + 'echo "bm-e2e-bash-version=$BASH_VERSION" >&2', + 'exec bash "$1"', + "", + ] + ), + encoding="utf-8", + newline="\n", + ) + cwd = work / "cwd" + cwd.mkdir(exist_ok=True) + + before = _pyvenv_state() + assert BASH is not None # pytestmark skipif guarantees this + proc = subprocess.run( # noqa: S603 — test harness, fixed argv + [BASH, _posix(driver), _posix(SETUP_SH)], + input=stdin, + stdin=subprocess.DEVNULL if stdin is None else None, + env=env, + cwd=str(cwd), + capture_output=True, + text=True, + timeout=timeout, + ) + after = _pyvenv_state() + assert after == before, ( + "CONTAINMENT BREACH: scripts/setup.sh modified the real repo " + ".venv/pyvenv.cfg — UV_PROJECT_ENVIRONMENT redirect is broken" + ) + + match = re.search(r"bm-e2e-bash-version=(\S+)", proc.stderr) + return SetupRun( + rc=proc.returncode, + stdout=proc.stdout, + stderr=proc.stderr, + home=home, + cwd=cwd, + bash_version=match.group(1) if match else "unknown", + ) + + +# --------------------------------------------------------------------------- +# Shared uv containment + the single uv-sync-bearing B1 run +# --------------------------------------------------------------------------- + + +@pytest.fixture(scope="module") +def uv_pins(tmp_path_factory: pytest.TempPathFactory) -> dict[str, str]: + """Env pins shared by every setup.sh run in this module. + + One tmp venv for all runs: the first ``uv sync`` materializes it from + the warm host cache; every later run (B2-B4) sees an up-to-date venv and + syncs in ~1s. Keeps total module runtime sane while still never touching + the real repo .venv. + """ + base = tmp_path_factory.mktemp("uv-shared") + return { + "UV_PROJECT_ENVIRONMENT": str(base / "venv"), + "UV_PYTHON": _base_python(), # never execute PATH python shims + **_host_uv_dirs(), + } + + +@pytest.fixture(scope="module") +def decline_run( + tmp_path_factory: pytest.TempPathFactory, uv_pins: dict[str, str] +) -> SetupRun: + """B1: ONE headless decline-flow run shared by all B1 assertions. + + Mirrors the ``clean_slate_home`` fixture shape (home is a subdirectory + so side artifacts stay outside the fake home) at module scope — the + function-scoped fixture cannot back a shared run. + """ + work = tmp_path_factory.mktemp("b1-decline") + home = work / "home" + home.mkdir() + env = isolated_env(home, **uv_pins) + return _run_setup(env, home=home, work=work, stdin="n\n", ollama_shim=True) + + +# --------------------------------------------------------------------------- +# B1 — e2e-setup-headless-decline-completes +# --------------------------------------------------------------------------- + + +class TestHeadlessDeclineCompletes: + def test_exit_zero_and_progress_markers_in_order(self, decline_run: SetupRun) -> None: + """set-e-safe end-to-end completion, with stage markers localizing + any failure: uv stage before layout stage before install stage.""" + assert decline_run.rc == 0, decline_run.combined + out = decline_run.stdout + deps = out.find("Dependencies installed.") + layout = out.find("Runtime layout ready.") + assert deps != -1, f"uv stage marker missing:\n{decline_run.combined}" + assert layout != -1, f"layout stage marker missing:\n{decline_run.combined}" + assert deps < layout, "'Dependencies installed.' must precede 'Runtime layout ready.'" + assert "[install_hooks] Restart Claude Code" in out + assert any(ln == "[setup] Done." for ln in _clean(out).splitlines()) + + def test_ollama_branch_is_warn_and_continue(self, decline_run: SetupRun) -> None: + """The curl-exits-1 function shim forces the daemon-unreachable + warn-and-continue branch (setup.sh:165-172) deterministically on + every host — the pull (:175) is unreachable, so no network.""" + combined = decline_run.combined + assert "Ollama daemon not reachable" in combined + assert "After Ollama is running, re-run this script" in combined + assert "Embedding model pulled." not in combined + # Judged conditional mirror: if the prompt branch fired anyway + # (no-shim refactor), it must have warn-and-continued, not aborted. + if "Ollama not found" in combined: + assert "Ollama still missing" in combined + + def test_runtime_layout_created(self, decline_run: SetupRun) -> None: + """setup.sh:188-191 — including the knowledge-base dirs the MCP + server never auto-creates (product bug #9's install-time half).""" + bm_home = decline_run.home / ".better-memory" + assert (bm_home / "spool").is_dir() + for sub in ("standards", "languages", "projects"): + assert (bm_home / "knowledge-base" / sub).is_dir(), sub + + def test_claude_json_command_is_win_path_correct(self, decline_run: SetupRun) -> None: + """The regression this file exists for: win_path (setup.sh:42-54) + must hand Claude Code a native Windows path, never MSYS /c/... form. + """ + config = json.loads((decline_run.home / ".claude.json").read_text(encoding="utf-8")) + server = config["mcpServers"]["better-memory"] + command = server["command"] + ctx = f"command={command!r} (bash {decline_run.bash_version})" + assert Path(command) == VENV_PY, ctx + if _WIN: + # Uppercase drive + backslashes only. Path() equality above + # already tolerates the bash<5.2 patsub_replacement double + # backslash form (Path collapses repeated separators); these + # pins reject the MSYS and mixed-separator forms outright. + assert re.match(r"^[A-Z]:", command), ctx + assert "/" not in command, ctx + assert "\\" in command, ctx + assert server["args"] == ["-m", "better_memory.mcp"] + assert Path(server["env"]["BETTER_MEMORY_HOME"]) == decline_run.home / ".better-memory" + + def test_settings_hooks_py_pyw_split(self, decline_run: SetupRun) -> None: + """Exactly 5 managed hook entries; needs_stdout hooks keep python.exe + (pythonw silently nulls stdout -> no additionalContext), background + hooks get pythonw.exe. The split must survive bash -> argparse.""" + settings = json.loads( + (decline_run.home / ".claude" / "settings.json").read_text(encoding="utf-8") + ) + entries: dict[tuple[str, str], str] = {} + matchers: dict[tuple[str, str], str | None] = {} + for event, groups in settings["hooks"].items(): + for group in groups: + for hook in group.get("hooks", []): + command = hook.get("command", "") + if "better_memory.hooks." not in command: + continue + parsed = re.fullmatch(r'"([^"]+)" -m ([\w.]+)', command) + assert parsed, f"unquoted interpreter or format drift: {command!r}" + key = (event, parsed.group(2)) + assert key not in entries, f"duplicate hook entry {key}" + entries[key] = parsed.group(1) + matchers[key] = group.get("matcher") + + expected_interp = { + ("SessionStart", "better_memory.hooks.session_bootstrap"): VENV_PY, + ("PostToolUse", "better_memory.hooks.observer"): VENV_PYW, + ("Stop", "better_memory.hooks.session_close"): VENV_PYW, + ("UserPromptSubmit", "better_memory.hooks.contextual_inject"): VENV_PY, + ("PreToolUse", "better_memory.hooks.contextual_inject"): VENV_PY, + } + assert set(entries) == set(expected_interp) # exactly the 5 managed entries + for key, interp in entries.items(): + assert Path(interp) == expected_interp[key], (key, interp) + assert matchers[("PostToolUse", "better_memory.hooks.observer")] == "Write|Edit|Bash" + assert ( + matchers[("PreToolUse", "better_memory.hooks.contextual_inject")] + == "Skill|Task|Write" + ) + + def test_all_writes_landed_under_tmp(self, decline_run: SetupRun) -> None: + """Containment: the fake home holds exactly the install products, the + working dir caught no stray relative writes, and (asserted inside + _run_setup on every run) the real repo .venv is untouched.""" + top = {p.name for p in decline_run.home.iterdir()} + required = {".better-memory", ".claude", ".claude.json"} + assert required <= top, top + # Host-toolchain noise under the redirected profile is tolerated — + # e.g. Windows Python 3.14's install manager regenerates Start Menu + # shortcuts under a fresh USERPROFILE (observed empirically). It is + # still under tmp, which is the containment contract. But nothing + # product-shaped may appear beyond the three install targets. + extras = top - required + leaks = {e for e in extras if "claude" in e.lower() or "memory" in e.lower()} + assert not leaks, f"unexpected product artifacts in fake home: {leaks}" + # setup.sh/install_hooks must not create any database (A1's zero-DB + # pin, re-checked here because this run went through the full script). + assert not list(decline_run.home.rglob("*.db")) + bm = {p.name for p in (decline_run.home / ".better-memory").iterdir()} + # Fresh slate: no pre-existing configs, so no install-backups/. + assert bm == {"spool", "knowledge-base"}, bm + # Stray-relative-write canary: setup.sh must not write product + # artifacts relative to the invoker's cwd. Same host-noise filter: + # pymanager also materializes a Python/bin shim dir into cwd when + # LOCALAPPDATA is absent from the env (observed empirically). + cwd_names = {p.name for p in decline_run.cwd.iterdir()} + cwd_leaks = { + n for n in cwd_names if "claude" in n.lower() or "memory" in n.lower() + } + assert not cwd_leaks, f"setup.sh wrote relative to cwd: {cwd_leaks}" + assert not list(decline_run.cwd.rglob("*.db")) + assert not list(decline_run.cwd.rglob("*.json")) + + +# --------------------------------------------------------------------------- +# B2 — e2e-setup-eof-aborts-all-or-nothing +# --------------------------------------------------------------------------- + +#: Replicates setup.sh:124-127's two ollama detection paths exactly — the +#: hardcoded fallback uses the *real* whoami and ignores HOME redirection, +#: so it must be probed, not assumed. +_OLLAMA_PROBE = """ +if command -v ollama >/dev/null 2>&1; then echo bm-e2e-ollama=found; exit 0; fi +if [[ -x "/c/Users/$(whoami)/AppData/Local/Programs/Ollama/ollama.exe" ]]; then + echo bm-e2e-ollama=found; exit 0 +fi +echo bm-e2e-ollama=absent +""" + + +def _ollama_free_path() -> str: + """The outer PATH minus any segment mentioning ollama.""" + segments = os.environ.get("PATH", "").split(os.pathsep) + return os.pathsep.join(s for s in segments if "ollama" not in s.lower()) + + +class TestEofAbortsAllOrNothing: + def test_eof_abort_is_all_or_nothing( + self, tmp_path: Path, uv_pins: dict[str, str] + ) -> None: + """Product bug #11 pinned: closed stdin dies at the Ollama read -rp + (setup.sh:134/141/148) under set -euo pipefail, BEFORE the layout + stage (:188) and install_hooks (:199). Flips red if the stages are + reordered ('do the important stuff first') or the prompt gains an + EOF-safe default — both are deliberate contract changes. + + Expected to execute only on ollama-free hosts (CI images); self-skips + on dev machines with Ollama installed, e.g. this repo's primary one. + """ + home = tmp_path / "home" + home.mkdir() + env = isolated_env(home, PATH=_ollama_free_path(), **uv_pins) + + assert BASH is not None + probe = subprocess.run( # noqa: S603 + [BASH, "-c", _OLLAMA_PROBE], env=env, capture_output=True, text=True, timeout=60 + ) + if "bm-e2e-ollama=found" in probe.stdout: + pytest.skip( + "host ollama detectable via setup.sh's probes (PATH or the " + "hardcoded whoami path) — the prompt/abort branch is " + "unreachable here; this test runs on ollama-free CI" + ) + assert "bm-e2e-ollama=absent" in probe.stdout, probe.stderr + + result = _run_setup(env, home=home, work=tmp_path, stdin=None, ollama_shim=False) + + assert result.rc == 1, result.combined + # Positive-progress markers: a death in the python/uv stages would + # also exit 1 with no config writes — these pin WHERE it died. + assert "Dependencies installed." in result.stdout, result.combined + assert "Ollama not found" in result.stderr, result.combined + # All-or-nothing: nothing was written before the abort. + assert not (home / ".claude.json").exists() + assert not (home / ".claude" / "settings.json").exists() + assert not (home / ".better-memory" / "spool").exists() + assert "Runtime layout ready" not in result.combined + assert "[install_hooks]" not in result.combined + + +# --------------------------------------------------------------------------- +# B3 — e2e-setup-default-home-derivation +# --------------------------------------------------------------------------- + + +class TestDefaultHomeDerivation: + def test_default_home_derived_and_propagated( + self, tmp_path: Path, uv_pins: dict[str, str] + ) -> None: + """BETTER_MEMORY_HOME deliberately ABSENT: the ${VAR:-default} + fallback (setup.sh:37-38) must derive $HOME/.better-memory and plumb + it via --home into the mcpServers env block and the backup location. + Invisible while every other test exports the var — this is the true + brand-new-user path. + + A trivial pre-seeded .claude.json makes run 1 produce a backup, so + the backup-location assertion needs no second uv-bearing run. + """ + home = tmp_path / "home" + home.mkdir() + seeded = json.dumps({"theme": "dark"}, indent=2) + "\n" + (home / ".claude.json").write_text(seeded, encoding="utf-8") + + env = isolated_env(home, BETTER_MEMORY_HOME=None, **uv_pins) + assert not any(k.upper() == "BETTER_MEMORY_HOME" for k in env) + result = _run_setup(env, home=home, work=tmp_path, stdin="n\n", ollama_shim=True) + + assert result.rc == 0, result.combined + bm_home = home / ".better-memory" + assert (bm_home / "spool").is_dir() + for sub in ("standards", "languages", "projects"): + assert (bm_home / "knowledge-base" / sub).is_dir(), sub + + config = json.loads((home / ".claude.json").read_text(encoding="utf-8")) + assert config["theme"] == "dark" # foreign top-level key preserved + value = config["mcpServers"]["better-memory"]["env"]["BETTER_MEMORY_HOME"] + # Platform-correct rendering: bash hands --home through MSYS + # argument conversion, so compare as paths, not strings. + assert Path(value) == bm_home, f"env BETTER_MEMORY_HOME={value!r}" + if _WIN: + assert not value.startswith("/"), f"MSYS-form path leaked: {value!r}" + + backups = sorted((bm_home / "install-backups").iterdir()) + assert [_backup_kind(p) for p in backups] == [".claude.json"] + assert backups[0].read_text(encoding="utf-8") == seeded + + +def _backup_kind(path: Path) -> str: + """'.claude.json.20260712-093000.bak' -> '.claude.json' (name format pin).""" + match = re.fullmatch(r"(?s)(.+)\.\d{8}-\d{6}\.bak", path.name) + assert match, f"backup name format drift: {path.name}" + return match.group(1) + + +# --------------------------------------------------------------------------- +# B4 — e2e-setup-install-hooks-failure-propagates +# --------------------------------------------------------------------------- + + +class TestInstallHooksFailurePropagates: + def test_failure_surfaces_with_remediation_and_stage_order( + self, tmp_path: Path, uv_pins: dict[str, str] + ) -> None: + """setup.sh:199-206's `|| { error ...; exit 1; }` must surface an + install_hooks failure — an `|| true` or dropped subshell status would + tell the user 'Done.' while their install silently failed. Also pins + stage order (layout precedes install: spool EXISTS on failure) and + validate-before-write through the bash wrapper (seed untouched, no + backups, no settings.json).""" + home = tmp_path / "home" + home.mkdir() + malformed = '{"mcpServers": {broken' + (home / ".claude.json").write_text(malformed, encoding="utf-8") + + env = isolated_env(home, **uv_pins) + result = _run_setup(env, home=home, work=tmp_path, stdin="n\n", ollama_shim=True) + + assert result.rc == 1, result.combined + assert "install_hooks failed" in result.stderr + assert "scripts/setup.sh aborting" in result.stderr + # The underlying : diagnostic from install_hooks' + # _load_or_empty must reach the user through the bash wrapper. + assert re.search(r"\.claude\.json:1:", result.combined), result.combined + assert "Fix the file then re-run" in result.combined + assert not any(ln == "[setup] Done." for ln in result.lines()) + + # Stage order pin: layout (setup.sh:188-191) ran before the failure. + bm_home = home / ".better-memory" + assert (bm_home / "spool").is_dir() + # Validate-before-write held: nothing was modified or backed up. + assert (home / ".claude.json").read_text(encoding="utf-8") == malformed + assert not (home / ".claude" / "settings.json").exists() + assert not (bm_home / "install-backups").exists() + assert not list(home.glob("**/*.tmp")) diff --git a/tests/e2e/test_sqlite_journey.py b/tests/e2e/test_sqlite_journey.py new file mode 100644 index 0000000..fe563fa --- /dev/null +++ b/tests/e2e/test_sqlite_journey.py @@ -0,0 +1,725 @@ +"""E2E sqlite journey (design section C, tasks C1-C6). + +Hermetic clean-slate smoke tests for the sqlite-mode new-user journey, +per ``docs/superpowers/specs/2026-07-12-e2e-clean-slate-smoke-design.md``: + +* C1 ``e2e-sqlite-first-boot-migrates-tools-knowledge`` +* C2 ``e2e-sqlite-hook-before-server-degraded`` (KNOWN-DEFECT PIN) +* C3 ``e2e-sqlite-hook-first-then-server-heals`` +* C4 ``e2e-sqlite-observe-retrieve-record-use-offline`` +* C5 ``e2e-sqlite-session-close-rate-then-marker`` +* C6 ``e2e-sqlite-spool-drain-synthesize-loop`` + +Deviation from the design's file split: the degraded/heal hook scenarios +(C2, C3) live HERE rather than in ``test_hooks_contracts.py`` — this module +is single-owner for the whole sqlite journey while another task owns the +contextual-inject contracts. + +Every subprocess env is built through :func:`tests.e2e._env.isolated_env` +(the single choke point); server spawns go through +:func:`tests.e2e.conftest.mcp_session`; hook spawns through ``run_hook``. +""" + +from __future__ import annotations + +import json +import sqlite3 +import time +from contextlib import closing +from pathlib import Path +from typing import Any + +import pytest + +from tests.e2e._env import isolated_env +from tests.e2e.conftest import mcp_session, run_hook + +BOOTSTRAP_HOOK = "better_memory.hooks.session_bootstrap" +CLOSE_HOOK = "better_memory.hooks.session_close" + +#: Wire-visible sqlite-mode tool surface (subset — never exact equality). +#: Includes BOTH synthesize tools: they are gated on +#: ``backend.supports_synthesis`` and must be advertised in sqlite mode. +EXPECTED_TOOLS: frozenset[str] = frozenset( + { + "memory.observe", + "memory.retrieve", + "memory.retrieve_observations", + "memory.record_use", + "memory.session_bootstrap", + "memory.start_episode", + "memory.close_episode", + "memory.list_session_exposures", + "memory.apply_session_ratings", + "memory.synthesize_next_get_context", + "memory.synthesize_next_apply", + "knowledge.search", + "knowledge.list", + } +) + +#: memory.db schema superset after migrations (subset assertion — a new +#: migration adding tables must not break this suite). +EXPECTED_MEMORY_TABLES: frozenset[str] = frozenset( + { + "observations", + "episodes", + "hook_errors", + "session_memory_exposure", + "reflections", + "semantic_memories", + } +) + + +# --------------------------------------------------------------------------- +# Local wire/db helpers +# --------------------------------------------------------------------------- + + +def _single_json_dict(result: Any) -> dict[str, Any]: + """Extract + parse the single TextContent payload of a successful call.""" + content = result.content + assert not getattr(result, "isError", False), f"tool errored: {content!r}" + assert len(content) == 1, f"expected one content block: {content!r}" + block = content[0] + assert getattr(block, "type", None) == "text", f"not a text block: {block!r}" + parsed = json.loads(block.text) + assert isinstance(parsed, dict), f"expected JSON object, got {block.text!r}" + return parsed + + +def _single_json_list(result: Any) -> list[Any]: + content = result.content + assert not getattr(result, "isError", False), f"tool errored: {content!r}" + assert len(content) == 1, f"expected one content block: {content!r}" + block = content[0] + assert getattr(block, "type", None) == "text", f"not a text block: {block!r}" + parsed = json.loads(block.text) + assert isinstance(parsed, list), f"expected JSON array, got {block.text!r}" + return parsed + + +def _error_text(result: Any) -> str: + """Assert the call failed at the MCP layer and return the error text.""" + content = result.content + assert getattr(result, "isError", False), f"expected isError, got: {content!r}" + assert len(content) >= 1, "error result carried no content blocks" + block = content[0] + assert getattr(block, "type", None) == "text", f"not a text block: {block!r}" + return block.text + + +def _hook_envelope(stdout: str) -> dict[str, Any]: + """Parse a hook's stdout as exactly one JSON envelope line.""" + lines = [line for line in stdout.splitlines() if line.strip()] + assert len(lines) == 1, f"expected exactly one JSON line on stdout: {stdout!r}" + parsed = json.loads(lines[0]) + assert isinstance(parsed, dict) + return parsed + + +def _ro_connect(db_path: Path) -> sqlite3.Connection: + """Read-only sqlite connection (post-exit inspection only).""" + conn = sqlite3.connect(f"file:{db_path.as_posix()}?mode=ro", uri=True) + conn.row_factory = sqlite3.Row + return conn + + +def _table_names(db_path: Path) -> set[str]: + assert db_path.exists(), f"database file missing: {db_path}" + with closing(_ro_connect(db_path)) as conn: + rows = conn.execute( + "SELECT name FROM sqlite_master WHERE type='table'" + ).fetchall() + return {r["name"] for r in rows} + + +def _query_one(db_path: Path, sql: str, params: tuple = ()) -> sqlite3.Row | None: + with closing(_ro_connect(db_path)) as conn: + return conn.execute(sql, params).fetchone() + + +def _session_end_markers(spool_dir: Path) -> list[Path]: + """Top-level (NON-recursive) session_end markers. + + Deliberately non-recursive: SpoolService.drain quarantines malformed + files into a subdirectory; a recursive glob would wrongly count those. + """ + if not spool_dir.is_dir(): + return [] + return sorted(spool_dir.glob("*_session_end_*.json")) + + +def _bootstrap_payload(proj_dir: Path, session_id: str = "e2e-session-1") -> dict: + return {"source": "startup", "session_id": session_id, "cwd": str(proj_dir)} + + +# --------------------------------------------------------------------------- +# C1 — e2e-sqlite-first-boot-migrates-tools-knowledge +# --------------------------------------------------------------------------- + + +async def test_first_boot_migrates_tools_knowledge( + clean_slate_home: Path, tmp_path: Path +) -> None: + """First ``python -m better_memory.mcp`` on a NONEXISTENT home: + initialize succeeds fully offline, the sqlite tool surface (incl. both + synthesize tools) is advertised, both DBs are created + migrated, the + knowledge tools return well-formed empty results on the unindexed + corpus, and ``knowledge-base/`` is NOT auto-created. + + KNOWN-DEFECT PIN (design §4 item 9): the server never mkdirs + ``knowledge-base/`` — pip installs get silently-empty knowledge search. + Flip the final assertion when the product decides to auto-create it. + """ + env = isolated_env(clean_slate_home) + bm_home = clean_slate_home / ".better-memory" + assert not bm_home.exists(), "precondition: virgin home, no .better-memory" + + with (tmp_path / "c1.stderr").open("w", encoding="utf-8") as errlog: + async with mcp_session(env, errlog=errlog) as session: + listed = await session.list_tools() + names = {tool.name for tool in listed.tools} + assert EXPECTED_TOOLS <= names, ( + f"missing tools: {sorted(EXPECTED_TOOLS - names)}" + ) + + # Folded gap-3: knowledge tools on a fresh, unindexed corpus + # return empty arrays — never errors. This is the mandated + # startup knowledge_search path of every fresh install. + search_hits = _single_json_list( + await session.call_tool( + "knowledge.search", {"query": "anything at all"} + ) + ) + assert search_hits == [] + docs = _single_json_list(await session.call_tool("knowledge.list", {})) + assert docs == [] + + # Post-exit inspection (no WAL-concurrency ambiguity). + memory_tables = _table_names(bm_home / "memory.db") + assert EXPECTED_MEMORY_TABLES <= memory_tables, ( + f"missing tables: {sorted(EXPECTED_MEMORY_TABLES - memory_tables)}" + ) + assert any("trigram" in name for name in memory_tables), ( + f"no trigram FTS table (migration 0011) in: {sorted(memory_tables)}" + ) + knowledge_tables = _table_names(bm_home / "knowledge.db") + assert len(knowledge_tables) >= 1 + + # DEFECT PIN: server does not auto-create the knowledge-base tree. + assert not (bm_home / "knowledge-base").exists(), ( + "knowledge-base/ was auto-created — the documented gap (design §4 " + "item 9) has been fixed; update this pin deliberately" + ) + + +# --------------------------------------------------------------------------- +# C2 — e2e-sqlite-hook-before-server-degraded (KNOWN-DEFECT PIN) +# --------------------------------------------------------------------------- + + +def test_hook_before_server_degraded( + clean_slate_home: Path, tmp_path: Path +) -> None: + """KNOWN-DEFECT PIN (design §4 item 3): SessionStart hook fired before + any server boot on a virgin home. Contract: exit 0, empty stderr, a + single fallback-directive envelope, a schema-less ``memory.db`` (the + file exists, 0 tables), NO ``runtime/sessions`` marker, no other dirs, + and the ``hook_errors`` write silently no-ops (no table to insert into). + + This test flips red on the intended fix (hook-side migrations or a + hoisted ``write_session_id``) — that is deliberate: the fix must update + this contract and the heal-sequence test together. + """ + env = isolated_env(clean_slate_home) + proj_dir = tmp_path / "proj" + proj_dir.mkdir() + bm_home = clean_slate_home / ".better-memory" + + rc, out, err = run_hook(BOOTSTRAP_HOOK, _bootstrap_payload(proj_dir), env) + + assert rc == 0 + assert err == "" + envelope = _hook_envelope(out) + assert set(envelope) == {"hookSpecificOutput"} + hso = envelope["hookSpecificOutput"] + assert hso["hookEventName"] == "SessionStart" + ctx = hso["additionalContext"] + # Prefix match — the table name is coupled to internal query order. + assert ctx.startswith( + "better-memory: session bootstrap failed " + "(OperationalError: no such table:" + ), f"unexpected fallback directive: {ctx!r}" + assert "Call mcp__better-memory__memory_session_bootstrap manually" in ctx + + # connect() created the file but never migrated it. + db_path = bm_home / "memory.db" + assert db_path.exists() + assert _table_names(db_path) == set(), "hook must not run migrations" + + # Marker skipped: write_session_id sits after bootstrap() in the try. + assert not (bm_home / "runtime").exists() + assert not (bm_home / "spool").exists() + # No other dirs at all; only the db file (+ possible WAL side files). + assert [p for p in bm_home.iterdir() if p.is_dir()] == [] + assert {p.name for p in bm_home.iterdir()} <= { + "memory.db", + "memory.db-wal", + "memory.db-shm", + } + + +# --------------------------------------------------------------------------- +# C3 — e2e-sqlite-hook-first-then-server-heals +# --------------------------------------------------------------------------- + + +async def test_hook_first_then_server_heals( + clean_slate_home: Path, tmp_path: Path +) -> None: + """The real new-user SEQUENCE: degraded SessionStart hook leaves a + schema-less WAL ``memory.db`` → the first server boot heals it + (migrations apply onto the pre-existing 0-table file) → the fallback + directive's promised ``memory.session_bootstrap`` MCP tool works + (``episode.action == 'opened'``) → a re-run of the hook reports + ``Episode: reused`` and writes the session marker. Exactly one open + episode row exists throughout. + """ + env = isolated_env(clean_slate_home) + proj_dir = tmp_path / "proj" + proj_dir.mkdir() + bm_home = clean_slate_home / ".better-memory" + payload = _bootstrap_payload(proj_dir) + + # Leg 1 — degraded hook on the virgin home (C2's core assertions). + rc, out, err = run_hook(BOOTSTRAP_HOOK, payload, env) + assert rc == 0, err + ctx = _hook_envelope(out)["hookSpecificOutput"]["additionalContext"] + assert ctx.startswith("better-memory: session bootstrap failed (") + db_path = bm_home / "memory.db" + assert _table_names(db_path) == set() + assert not (bm_home / "runtime" / "sessions").exists() + + # Leg 2 — server boots onto the pre-existing 0-table WAL file and + # heals it; the directive's remediation tool actually works. + with (tmp_path / "c3.stderr").open("w", encoding="utf-8") as errlog: + async with mcp_session(env, errlog=errlog) as session: + result = _single_json_dict( + await session.call_tool( + "memory.session_bootstrap", + {"session_id": "e2e-session-1", "cwd": str(proj_dir)}, + ) + ) + assert result["episode"]["action"] == "opened", result + assert result["episode"]["id"] + + # Migrations applied onto the previously schema-less file. + memory_tables = _table_names(db_path) + assert EXPECTED_MEMORY_TABLES <= memory_tables + assert any("trigram" in name for name in memory_tables) + + # Leg 3 — hook re-run now succeeds: episode reused, marker written. + rc, out, err = run_hook(BOOTSTRAP_HOOK, payload, env) + assert rc == 0, err + ctx = _hook_envelope(out)["hookSpecificOutput"]["additionalContext"] + assert "Episode: reused" in ctx, f"expected reuse, got: {ctx!r}" + + sessions_dir = bm_home / "runtime" / "sessions" + marker_files = [p for p in sessions_dir.iterdir() if p.is_file()] + assert len(marker_files) == 1, ( + f"expected exactly one marker (no .sid-* droppings): {marker_files}" + ) + assert marker_files[0].read_text(encoding="utf-8").strip() == "e2e-session-1" + + # Episodes bind to sessions via the episode_sessions join table + # (an episode may span multiple sessions — migration 0002). + row = _query_one( + db_path, + "SELECT COUNT(DISTINCT e.id) AS n FROM episodes e " + "JOIN episode_sessions es ON es.episode_id = e.id " + "WHERE es.session_id = ? AND e.ended_at IS NULL", + ("e2e-session-1",), + ) + assert row is not None and row["n"] == 1, ( + "exactly one open background episode expected across tool call + re-run" + ) + + +# --------------------------------------------------------------------------- +# C4 — e2e-sqlite-observe-retrieve-record-use-offline +# --------------------------------------------------------------------------- + + +async def test_observe_retrieve_record_use_offline( + clean_slate_home: Path, tmp_path: Path +) -> None: + """Full offline data loop in ONE server session: observe → trigram + retrieve (the obs id surfaces) → bucket shape ``{do,dont,neutral}`` → + ``record_use(success)`` on A and ``record_use(failure)`` on B → + not-found id is an MCP isError. Durability is proven by POST-EXIT DB + reads (a dropped commit passes any same-connection unit test but loses + the ratings at process exit). + + The <30s wall-clock budget around the observe call is the tripwire for + a regression reintroducing the Ollama embed path (3 retries against the + poisoned host blow past it or raise EmbeddingError). + """ + env = isolated_env(clean_slate_home) + bm_home = clean_slate_home / ".better-memory" + + with (tmp_path / "c4.stderr").open("w", encoding="utf-8") as errlog: + async with mcp_session(env, errlog=errlog) as session: + t0 = time.monotonic() + obs_a = _single_json_dict( + await session.call_tool( + "memory.observe", + { + "content": ( + "Zebra flux capacitor timed out during spool " + "drain on Windows" + ), + "outcome": "failure", + "component": "spool", + "theme": "bug", + }, + ) + )["id"] + observe_elapsed = time.monotonic() - t0 + assert observe_elapsed < 30, ( + f"observe took {observe_elapsed:.1f}s — sqlite embeddings " + "bypass lost? (Ollama retry path reintroduced)" + ) + assert isinstance(obs_a, str) and obs_a + + obs_b = _single_json_dict( + await session.call_tool( + "memory.observe", + { + "content": "quiet baseline observation for failure leg", + "outcome": "neutral", + }, + ) + )["id"] + + # Trigram FTS populated synchronously by the insert trigger. + rows = _single_json_list( + await session.call_tool( + "memory.retrieve_observations", + {"query": "flux capacitor timed out"}, + ) + ) + by_id = {r["id"]: r for r in rows} + assert obs_a in by_id, f"trigram retrieval missed {obs_a}: {rows}" + assert "flux capacitor" in by_id[obs_a]["content"] + assert by_id[obs_a]["outcome"] == "failure" + + # Bucket shape contract every fresh install's CLAUDE.md relies on. + buckets = _single_json_dict( + await session.call_tool("memory.retrieve", {}) + ) + assert {"do", "dont", "neutral"} <= set(buckets) + + # Both reinforcement branches + the not-found contract. + ok_a = _single_json_dict( + await session.call_tool( + "memory.record_use", {"id": obs_a, "outcome": "success"} + ) + ) + assert ok_a == {"ok": True} + ok_b = _single_json_dict( + await session.call_tool( + "memory.record_use", {"id": obs_b, "outcome": "failure"} + ) + ) + assert ok_b == {"ok": True} + + not_found = await session.call_tool( + "memory.record_use", + {"id": "does-not-exist", "outcome": "success"}, + ) + assert "not found" in _error_text(not_found).lower() + + # Post-exit durability: re-open the DB after the server process died. + db_path = bm_home / "memory.db" + row_a = _query_one( + db_path, + "SELECT used_count, validated_true, validated_false, " + "reinforcement_score, last_used, last_validated " + "FROM observations WHERE id = ?", + (obs_a,), + ) + assert row_a is not None + assert row_a["used_count"] == 1 + assert row_a["validated_true"] == 1 + assert row_a["validated_false"] == 0 + assert row_a["reinforcement_score"] == pytest.approx(1.0) + assert row_a["last_used"], "last_used must be stamped" + assert row_a["last_validated"], "last_validated must be stamped" + + row_b = _query_one( + db_path, + "SELECT used_count, validated_true, validated_false, " + "reinforcement_score FROM observations WHERE id = ?", + (obs_b,), + ) + assert row_b is not None + assert row_b["used_count"] == 1 + assert row_b["validated_true"] == 0 + assert row_b["validated_false"] == 1 + assert row_b["reinforcement_score"] == pytest.approx(-1.0) + + +# --------------------------------------------------------------------------- +# C5 — e2e-sqlite-session-close-rate-then-marker +# --------------------------------------------------------------------------- + + +async def test_session_close_rate_then_marker( + clean_slate_home: Path, tmp_path: Path +) -> None: + """Two-fire Stop sequence with a REAL rating turn between the fires. + + Fire 1 blocks with RATE_MEMORIES (memory id + skill name in the + directive) and writes ZERO session_end markers. The rating turn runs + against a server spawned with ``cwd=proj_dir`` and ``CLAUDE_SESSION_ID`` + DELETED — forcing the hook→marker-file→server session bridge + (``resolve_session_id`` falls back to the marker keyed by the server's + cwd, which must equal the payload cwd the bootstrap hook keyed on). + Fire 2 emits EMPTY stdout and exactly one session_end marker. + """ + env = isolated_env(clean_slate_home) + proj_dir = tmp_path / "proj" + proj_dir.mkdir() + bm_home = clean_slate_home / ".better-memory" + spool_dir = bm_home / "spool" + + # Seed one semantic memory through the real server (also migrates). + with (tmp_path / "c5-seed.stderr").open("w", encoding="utf-8") as errlog: + async with mcp_session(env, errlog=errlog) as session: + sem_id = _single_json_dict( + await session.call_tool( + "memory.semantic_observe", + {"content": "Always poison OLLAMA_HOST in hermetic tests"}, + ) + )["id"] + assert isinstance(sem_id, str) and sem_id + + # Bootstrap hook: creates the source='bootstrap' exposure row and + # writes the session marker keyed by the payload cwd (proj_dir). + rc, out, err = run_hook(BOOTSTRAP_HOOK, _bootstrap_payload(proj_dir), env) + assert rc == 0, err + boot_ctx = _hook_envelope(out)["hookSpecificOutput"]["additionalContext"] + assert "session bootstrap failed" not in boot_ctx, boot_ctx + assert "Episode:" in boot_ctx + + # Fire 1 — unrated exposure exists → decision:block, marker SKIPPED. + close_payload = {"session_id": "e2e-session-1", "cwd": str(proj_dir)} + rc, out, err = run_hook(CLOSE_HOOK, close_payload, env) + assert rc == 0, err + fire1 = _hook_envelope(out) + assert fire1["decision"] == "block" + assert "RATE_MEMORIES" in fire1["reason"] + assert fire1["hookSpecificOutput"]["hookEventName"] == "Stop" + directive = fire1["hookSpecificOutput"]["additionalContext"] + assert sem_id in directive + assert "rate-session-memories" in directive + assert _session_end_markers(spool_dir) == [], ( + "marker written while blocking — double session_end downstream" + ) + + # Rating turn — marker-bridge leg: CLAUDE_SESSION_ID deleted from the + # server env, server cwd pinned to proj_dir so the getcwd() fallback + # resolves the same marker key the hook wrote. + bridge_env = isolated_env(clean_slate_home, CLAUDE_SESSION_ID=None) + with (tmp_path / "c5-rate.stderr").open("w", encoding="utf-8") as errlog: + async with mcp_session(bridge_env, errlog=errlog, cwd=proj_dir) as session: + exposures_payload = _single_json_dict( + await session.call_tool("memory.list_session_exposures", {}) + ) + # Session id resolved FROM THE MARKER FILE — the bridge works. + assert exposures_payload["session_id"] == "e2e-session-1" + exposures = exposures_payload["exposures"] + assert exposures, "expected at least the seeded semantic exposure" + assert any( + e["kind"] == "semantic" and e["id"] == sem_id for e in exposures + ), exposures + for entry in exposures: + # Real wire shape: kind/id/title-or-content/exposed_at/source. + # There is NO rated_at on the wire (unrated filter is in SQL). + assert {"kind", "id", "exposed_at", "source"} <= set(entry) + assert "rated_at" not in entry + + # Exact payload shape the tool schema accepts: + # items require exactly {kind, id, class}. + ratings = [ + {"kind": e["kind"], "id": e["id"], "class": "ignored"} + for e in exposures + ] + applied_payload = _single_json_dict( + await session.call_tool( + "memory.apply_session_ratings", {"ratings": ratings} + ) + ) + assert sum(applied_payload["applied"].values()) >= 1, applied_payload + + # Fire 2 — everything rated: EMPTY stdout (stronger than substring + # absence: the marker path prints nothing at all), exactly one marker. + rc, out, err = run_hook(CLOSE_HOOK, close_payload, env) + assert rc == 0, err + assert out == "", f"fire 2 must print nothing, got: {out!r}" + markers = _session_end_markers(spool_dir) + assert len(markers) == 1, f"expected exactly one session_end marker: {markers}" + body = json.loads(markers[0].read_text(encoding="utf-8")) + assert body["event_type"] == "session_end" + assert body["session_id"] == "e2e-session-1" + + +# --------------------------------------------------------------------------- +# C6 — e2e-sqlite-spool-drain-synthesize-loop +# --------------------------------------------------------------------------- + + +async def test_spool_drain_synthesize_loop( + clean_slate_home: Path, tmp_path: Path +) -> None: + """session_end marker + next-session ``memory.retrieve`` drains the + spool and closes the background episode as ``no_outcome``; the + get_context/apply loop distills it into a reflection that the final + retrieve surfaces in the ``dont`` bucket. + + Session A: bootstrap hook opens background episode E → observe links + obs to E → Stop hook writes the marker immediately (nothing seeded, so + no RATE block). Session B (``CLAUDE_SESSION_ID=e2e-session-2`` + explicitly): retrieve drains + closes E; ``start_episode`` reports + ``pending_synthesis.pending >= 1``; get_context carries obs_id; apply + returns ``ok/counts.created==1/queue``; a second get_context excludes E. + """ + env = isolated_env(clean_slate_home) # CLAUDE_SESSION_ID=e2e-session-1 + proj_dir = tmp_path / "proj" + proj_dir.mkdir() + bm_home = clean_slate_home / ".better-memory" + spool_dir = bm_home / "spool" + + # Migrate the home (initialize-only boot). + with (tmp_path / "c6-migrate.stderr").open("w", encoding="utf-8") as errlog: + async with mcp_session(env, errlog=errlog): + pass + + # Session A: hook opens background episode E for e2e-session-1. + rc, out, err = run_hook(BOOTSTRAP_HOOK, _bootstrap_payload(proj_dir), env) + assert rc == 0, err + boot_ctx = _hook_envelope(out)["hookSpecificOutput"]["additionalContext"] + assert "session bootstrap failed" not in boot_ctx, boot_ctx + + # Observe inside session A — links to E via the active-episode lookup. + with (tmp_path / "c6-a.stderr").open("w", encoding="utf-8") as errlog: + async with mcp_session(env, errlog=errlog) as session: + obs_id = _single_json_dict( + await session.call_tool( + "memory.observe", + { + "content": "spool drain e2e observation about flux", + "outcome": "failure", + }, + ) + )["id"] + + # Stop hook: no unrated exposures → marker written immediately. + rc, out, err = run_hook( + CLOSE_HOOK, {"session_id": "e2e-session-1", "cwd": str(proj_dir)}, env + ) + assert rc == 0, err + assert out == "", f"no RATE block expected (nothing seeded): {out!r}" + assert len(_session_end_markers(spool_dir)) == 1 + + # Session B: a NEW session drains, synthesizes, and retrieves. + env_b = isolated_env(clean_slate_home, CLAUDE_SESSION_ID="e2e-session-2") + with (tmp_path / "c6-b.stderr").open("w", encoding="utf-8") as errlog: + async with mcp_session(env_b, errlog=errlog) as session: + # retrieve triggers SpoolService.drain: consumes the marker and + # closes background episode E as no_outcome. + buckets = _single_json_dict( + await session.call_tool("memory.retrieve", {}) + ) + assert {"do", "dont", "neutral"} <= set(buckets) + assert _session_end_markers(spool_dir) == [], ( + "session_end marker not consumed by the drain" + ) + + started = _single_json_dict( + await session.call_tool( + "memory.start_episode", {"goal": "next task"} + ) + ) + assert started["pending_synthesis"]["pending"] >= 1, started + + ctx = _single_json_dict( + await session.call_tool("memory.synthesize_next_get_context", {}) + ) + episode_e = ctx["episode_id"] + assert episode_e, f"queue empty — drain did not close E: {ctx}" + assert ctx["episode"]["goal"] is None, ( + "expected the BACKGROUND episode (goal NULL), not a foreground one" + ) + assert obs_id in {o["id"] for o in ctx["observations"]}, ctx + + decision = { + "new": [ + { + "title": "Pin flux drain timeouts", + "phase": "implementation", + "polarity": "dont", + "use_cases": "when draining spool in e2e", + "hints": ["always set a timeout"], + "confidence": 0.8, + "source_observation_ids": [obs_id], + } + ], + "augment": [], + "merge": [], + "ignore": [], + } + applied = _single_json_dict( + await session.call_tool( + "memory.synthesize_next_apply", + {"episode_id": episode_e, "decision": decision}, + ) + ) + assert applied["ok"] is True, applied + assert applied["counts"]["created"] == 1, applied + assert {"pending", "in_cooldown", "done"} <= set(applied["queue"]) + + # Mark-synthesized idempotency: E is never offered again. + ctx2 = _single_json_dict( + await session.call_tool("memory.synthesize_next_get_context", {}) + ) + assert ctx2["episode_id"] != episode_e + + # The loop closes: the new reflection surfaces for future sessions. + final = _single_json_dict( + await session.call_tool("memory.retrieve", {}) + ) + assert any( + r["title"] == "Pin flux drain timeouts" for r in final["dont"] + ), final + + # Post-exit DB: E closed by the drain, stamped by the apply. + db_path = bm_home / "memory.db" + episode_row = _query_one( + db_path, + "SELECT outcome, ended_at, synthesized_at FROM episodes WHERE id = ?", + (episode_e,), + ) + assert episode_row is not None + assert episode_row["outcome"] == "no_outcome" + assert episode_row["ended_at"], "drain must stamp ended_at" + assert episode_row["synthesized_at"], "apply must stamp synthesized_at" + + refl_count = _query_one( + db_path, + "SELECT COUNT(*) AS n FROM reflections WHERE title = ?", + ("Pin flux drain timeouts",), + ) + assert refl_count is not None and refl_count["n"] == 1 diff --git a/tests/e2e/test_sqlite_negative.py b/tests/e2e/test_sqlite_negative.py new file mode 100644 index 0000000..27448b7 --- /dev/null +++ b/tests/e2e/test_sqlite_negative.py @@ -0,0 +1,119 @@ +"""e2e-sqlite-ollama-absent-default-backend (design §1C row C8, task 5). + +The path a pip-install user actually hits: no Ollama installed, no +BETTER_MEMORY_EMBEDDINGS_BACKEND exported, so the DEFAULT backend +('ollama', config.py _DEFAULT_EMBEDDINGS_BACKEND) is in force and +OLLAMA_HOST points nowhere. Pins the documented partial-availability +contract: + +* server boot survives (startup probe is warn-only — mcp/server.py + _probe_ollama) with a clear stderr warning; +* memory.observe fails IN BAND (CallToolResult.isError, EmbeddingError + text from embeddings/ollama.py) — never a protocol crash; +* the failure is fast: <30s timed ONLY around the observe call; +* knowledge tools keep working; +* KNOWN-DEFECT PIN (design §4 item 10): the failed observe leaves an + orphan episode row behind. +""" + +from __future__ import annotations + +import sqlite3 +import time +from contextlib import closing +from pathlib import Path + +from tests.e2e._env import isolated_env +from tests.e2e.conftest import mcp_session + +#: Loopback discard port: no listener on dev/CI boxes, so connects fail with +#: an instant ECONNREFUSED — no DNS, no route, no 30s connect timeout. If a +#: machine ever runs something on port 9 the failure mode is a clear +#: assertion message, not a hang (design C8 determinism note). +OLLAMA_ABSENT_HOST = "http://127.0.0.1:9" + + +def _text_of(result: object) -> str: + return "".join( + getattr(block, "text", "") for block in getattr(result, "content", []) + ) + + +async def test_ollama_absent_default_backend_degrades_in_band( + clean_slate_home: Path, tmp_path: Path +) -> None: + """Default (ollama) embeddings + unreachable daemon: warn, degrade, persist. + + Regressions this flips red on (design C8): + * fatal startup probe — initialize() would raise, bricking every + Ollama-less install; + * EmbeddingError escaping as a server crash instead of isError=True; + * retry/timeout inflation (~90s stalls per observe) — the <30s budget; + * silent trigram auto-fallback (observe stops erroring — a deliberate + contract change that must be made consciously); + * the orphan-episode fix landing (see the pinned assertion below). + + KNOWN-DEFECT PIN — orphan episode on failed observe (design §4 item 10): + ObservationService.create commits a background episode in step 1, then + the embed in step 2 fails against the dead host, so the episode row is + orphaned (episodes == 1, observations == 0). When the product fix lands + (episode write deferred/rolled back on embed failure), DELETE the two + orphan-pin assertions at the bottom of this test — everything else here + is the enduring degradation contract. + """ + env = isolated_env( + clean_slate_home, + # Remove the harness default 'sqlite' pin: the whole point is the + # real out-of-the-box default ('ollama'). + BETTER_MEMORY_EMBEDDINGS_BACKEND=None, + OLLAMA_HOST=OLLAMA_ABSENT_HOST, + ) + errlog_path = tmp_path / "server.stderr" # outside the fake home + + with errlog_path.open("w", encoding="utf-8") as errlog: + async with mcp_session(env, errlog=errlog) as session: + # Boot survived (initialize happened inside mcp_session) and the + # embedding-dependent tool is still advertised. + listed = await session.list_tools() + assert "memory.observe" in {tool.name for tool in listed.tools} + + # Latency budget timed ONLY around the observe call — never the + # spawn or initialize (design/judge fix). Measured baseline on a + # warm Windows dev machine: 7.7s (3 connection-refused attempts, + # 1.5s total fixed backoff, MCP round-trip). 30s keeps loud + # failure on the ~90s retry-inflation regression this targets. + t0 = time.monotonic() + r_observe = await session.call_tool( + "memory.observe", {"content": "e2e negative-path observation"} + ) + observe_elapsed = time.monotonic() - t0 + + r_knowledge = await session.call_tool("knowledge.list", {}) + + assert observe_elapsed < 30.0, f"observe took {observe_elapsed:.1f}s (budget 30s)" + + # In-band tool error, not a protocol crash (ollama.py raises + # EmbeddingError('Failed to reach Ollama at after 3 attempts: ...')). + assert r_observe.isError is True + assert f"Failed to reach Ollama at {OLLAMA_ABSENT_HOST}" in _text_of(r_observe) + + # Partial-availability contract: knowledge tools are Ollama-independent. + assert not r_knowledge.isError, _text_of(r_knowledge) + + # Warn-only startup probe left its breadcrumb on stderr + # (mcp/server.py _probe_ollama; subset match on the host, not the + # full message). + stderr_text = errlog_path.read_text(encoding="utf-8", errors="replace") + assert f"Ollama unreachable at {OLLAMA_ABSENT_HOST}" in stderr_text + + # KNOWN-DEFECT PIN (see docstring): orphan episode committed, no + # observation row. Post-exit read — the server process is gone, so this + # also proves the episode commit was durable. + db = clean_slate_home / ".better-memory" / "memory.db" + with closing(sqlite3.connect(db)) as conn: + (episodes,) = conn.execute("SELECT COUNT(*) FROM episodes").fetchone() + (observations,) = conn.execute( + "SELECT COUNT(*) FROM observations" + ).fetchone() + assert episodes == 1 # DELETE when the orphan-episode defect is fixed + assert observations == 0 diff --git a/tests/e2e/test_tripwires.py b/tests/e2e/test_tripwires.py new file mode 100644 index 0000000..aad91d9 --- /dev/null +++ b/tests/e2e/test_tripwires.py @@ -0,0 +1,174 @@ +"""Network tripwires (design §1F). + +This module currently owns ONLY ``e2e-ollama-zero-traffic-tripwire`` +(design task 5). The AWS lockdown tripwire (``e2e-aws-lockdown-tripwire``) +is owned by the fake-agentcore task (design task 6) and is added to this +module separately — do not duplicate it here. + +Mechanism: point OLLAMA_HOST at a local recording HTTP server instead of the +usual ``.invalid`` poison, run a mini sqlite journey through the real MCP +server plus both synchronous hooks, then assert ZERO requests were recorded. +Strictly stronger than the poison host: it proves no attempt was ever made, +not merely that a failed attempt was survivable (and a poisoned attempt +retried 3x adds ~90s of latency that masquerades as flakiness). +""" + +from __future__ import annotations + +import json +import threading +import urllib.request +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path + +from tests.e2e._env import isolated_env +from tests.e2e.conftest import mcp_session, run_hook + + +class _RecordingHandler(BaseHTTPRequestHandler): + """Record every request under the server lock; always answer 200 '{}'.""" + + def _handle(self) -> None: + length = int(self.headers.get("Content-Length") or 0) + if length: + self.rfile.read(length) + server: _RecorderServer = self.server # type: ignore[assignment] + with server.lock: + server.requests.append((self.command, self.path)) + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.end_headers() + self.wfile.write(b"{}") + + do_GET = _handle + do_POST = _handle + do_HEAD = _handle + do_PUT = _handle + + def log_message(self, *args: object) -> None: # silence stderr chatter + pass + + +class _RecorderServer(ThreadingHTTPServer): + daemon_threads = True + + def __init__(self) -> None: + self.requests: list[tuple[str, str]] = [] + self.lock = threading.Lock() + # 127.0.0.1:0 — OS-assigned ephemeral port, no collisions, no DNS. + super().__init__(("127.0.0.1", 0), _RecordingHandler) + + +async def test_ollama_zero_traffic_across_journey_and_sync_hooks( + clean_slate_home: Path, tmp_path: Path +) -> None: + """e2e-ollama-zero-traffic-tripwire: the T1 journey never contacts Ollama. + + With BETTER_MEMORY_EMBEDDINGS_BACKEND=sqlite (the isolated_env default), + create_server must skip OllamaEmbedder construction AND the /api/tags + startup probe entirely (both live inside the ``embeddings_backend == + 'ollama'`` branch of mcp/server.py). Regressions this flips red on: + hoisting the probe out of that branch "for better diagnostics", or + contextual_inject growing an embedding-based scorer that ignores the + embeddings backend — either records a request here. + + Anti-vacuity (a dead server also makes zero requests): the recorder is + first proven to record via a direct self-check request, and the journey + itself carries positive assertions (observe returns an id, the trigram + drill-down surfaces it, retrieve returns well-formed buckets, both hooks + exit 0 with exactly one JSON envelope line). + """ + recorder = _RecorderServer() + thread = threading.Thread(target=recorder.serve_forever, daemon=True) + thread.start() + try: + port = recorder.server_address[1] + recorder_url = f"http://127.0.0.1:{port}" + + # --- recorder self-check: prove the tripwire can fire at all ------ + with urllib.request.urlopen( # noqa: S310 — loopback literal + recorder_url + "/__selfcheck", timeout=5 + ) as resp: + assert resp.status == 200 + with recorder.lock: + assert recorder.requests == [("GET", "/__selfcheck")] + recorder.requests.clear() + + env = isolated_env(clean_slate_home, OLLAMA_HOST=recorder_url) + proj_dir = tmp_path / "proj" + proj_dir.mkdir() + errlog_path = tmp_path / "server.stderr" # outside the fake home + + # --- mini journey through the real MCP server --------------------- + with errlog_path.open("w", encoding="utf-8") as errlog: + async with mcp_session(env, errlog=errlog) as session: + r_obs = await session.call_tool( + "memory.observe", + {"content": "tripwire observation", "outcome": "success"}, + ) + assert not r_obs.isError, r_obs.content + obs_id = json.loads(r_obs.content[0].text)["id"] + assert obs_id + + # Trigram drill-down (embedder is None in sqlite embeddings + # mode; migration 0011's insert triggers make this + # synchronous with the observe above). + r_drill = await session.call_tool( + "memory.retrieve_observations", {"query": "tripwire"} + ) + assert not r_drill.isError, r_drill.content + drilled = json.loads(r_drill.content[0].text) + matched = [o for o in drilled if o["id"] == obs_id] + assert matched, f"observation {obs_id} not surfaced: {drilled!r}" + assert "tripwire observation" in matched[0]["content"] + + # Bucketed retrieve stays well-formed (schema forbids extra + # args — reflections filters only, so call it bare). + r_buckets = await session.call_tool("memory.retrieve", {}) + assert not r_buckets.isError, r_buckets.content + buckets = json.loads(r_buckets.content[0].text) + assert {"do", "dont", "neutral"} <= set(buckets) + + # --- both synchronous hooks under the same recorder env ----------- + rc_boot, out_boot, err_boot = run_hook( + "better_memory.hooks.session_bootstrap", + { + "source": "startup", + "session_id": "e2e-session-1", + "cwd": str(proj_dir), + }, + env, + ) + assert rc_boot == 0, err_boot + boot_lines = out_boot.strip().splitlines() + assert len(boot_lines) == 1, out_boot + boot_envelope = json.loads(boot_lines[0]) + # DB was migrated by the server boot above, so bootstrap succeeds + # (positive journey assertion, not the degraded fallback). + assert "Episode:" in boot_envelope["hookSpecificOutput"]["additionalContext"] + + rc_inject, out_inject, err_inject = run_hook( + "better_memory.hooks.contextual_inject", + { + "hook_event_name": "UserPromptSubmit", + "prompt": "zero traffic tripwire prompt", + "session_id": "e2e-session-1", + "cwd": str(proj_dir), + }, + env, + ) + assert rc_inject == 0, err_inject + inject_lines = out_inject.strip().splitlines() + assert len(inject_lines) == 1, out_inject + assert "hookSpecificOutput" in json.loads(inject_lines[0]) + + # --- the tripwire itself ------------------------------------------- + with recorder.lock: + recorded = list(recorder.requests) + assert recorded == [], ( + "Ollama traffic detected on a sqlite-embeddings journey — the " + f"probe/embed path has become unconditional again: {recorded!r}" + ) + finally: + recorder.shutdown() + recorder.server_close() diff --git a/tests/e2e/test_tripwires_aws.py b/tests/e2e/test_tripwires_aws.py new file mode 100644 index 0000000..9762877 --- /dev/null +++ b/tests/e2e/test_tripwires_aws.py @@ -0,0 +1,199 @@ +"""e2e-aws-lockdown-tripwire (design section 1F). + +Proves the T2 harness can never touch real AWS: + +* helper contract — ``agentcore_env`` pins the endpoint to loopback, uses + fake static creds, locks the credential/config files onto nonexistent + tmp paths, disables IMDS, and lets NO outer ``AWS_*`` var survive; +* round trip — a real CLI subprocess (``better-memory agentcore status``) + against the fake: every request's Host is ``127.0.0.1:*`` and every + SigV4 scope is ``Credential=bm-e2e-fake/...`` (i.e. the default chain + never reached a real ``~/.aws``, SSO cache, IMDS, or shell creds) — + red BEFORE anything could ever be billed; +* workaround containment — the dummy ``BETTER_MEMORY_AGENTCORE_*_MEMORY_ID`` + literals (FIXME(idvar-gate) workaround) are grep-pinned to exactly two + files: the single env helper and the one defect-pin negative test. +""" + +from __future__ import annotations + +import subprocess +import sys +from pathlib import Path + +import pytest + +from tests.e2e._agentcore_env import ( + DUMMY_ID_VARS, + FAKE_ACCESS_KEY_ID, + FAKE_SECRET_ACCESS_KEY, + agentcore_env, + write_fake_agentcore_json, +) +from tests.e2e._fake_agentcore import FakeAgentCore, RecordedRequest + +#: The complete set of AWS_* keys the helper is allowed to emit. Anything +#: else (AWS_PROFILE, AWS_SESSION_TOKEN, AWS_IGNORE_CONFIGURED_ENDPOINT_URLS, +#: a real AWS_ACCESS_KEY_ID from the dev shell, ...) must never survive. +_LOCKDOWN_AWS_KEYS = frozenset( + { + "AWS_ENDPOINT_URL", + "AWS_ACCESS_KEY_ID", + "AWS_SECRET_ACCESS_KEY", + "AWS_SHARED_CREDENTIALS_FILE", + "AWS_CONFIG_FILE", + "AWS_EC2_METADATA_DISABLED", + } +) + + +class TestAgentcoreEnvLockdownContract: + """Pure helper-contract assertions (no subprocess).""" + + def test_lockdown_contract_under_hostile_outer_shell( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + # Simulate the most dangerous outer shell: real-looking creds, a + # profile, SSO/session material, and the endpoint-override kill + # switch. NONE of it may reach a child process. + monkeypatch.setenv("AWS_ACCESS_KEY_ID", "AKIAREALLOOKINGKEY") + monkeypatch.setenv("AWS_SECRET_ACCESS_KEY", "real-secret") + monkeypatch.setenv("AWS_PROFILE", "prod") + monkeypatch.setenv("AWS_SESSION_TOKEN", "real-session-token") + monkeypatch.setenv("AWS_WEB_IDENTITY_TOKEN_FILE", "/real/token") + monkeypatch.setenv("AWS_CONTAINER_CREDENTIALS_RELATIVE_URI", "/creds") + monkeypatch.setenv("AWS_IGNORE_CONFIGURED_ENDPOINT_URLS", "true") + monkeypatch.setenv("aws_region", "us-east-1") # case-insensitive drop + + env = agentcore_env(tmp_path / "home", 45999) + + aws_keys = {k for k in env if k.upper().startswith("AWS_")} + assert aws_keys == set(_LOCKDOWN_AWS_KEYS) + + assert env["AWS_ENDPOINT_URL"] == "http://127.0.0.1:45999" + assert env["AWS_ACCESS_KEY_ID"] == FAKE_ACCESS_KEY_ID + assert env["AWS_SECRET_ACCESS_KEY"] == FAKE_SECRET_ACCESS_KEY + assert env["AWS_EC2_METADATA_DISABLED"] == "true" + # Credential-chain files locked onto nonexistent-by-construction paths. + for key in ("AWS_SHARED_CREDENTIALS_FILE", "AWS_CONFIG_FILE"): + locked = Path(env[key]) + assert not locked.exists() + assert Path(str(tmp_path)) in locked.parents + + assert env["BETTER_MEMORY_STORAGE_BACKEND"] == "agentcore" + assert env["BETTER_MEMORY_AGENTCORE_REGION"] + # FIXME(idvar-gate) workaround values present (see _agentcore_env). + for var_name, dummy_value in DUMMY_ID_VARS.items(): + assert env[var_name] == dummy_value + + def test_pins_can_remove_lockdown_keys_for_negative_tests( + self, tmp_path: Path + ) -> None: + env = agentcore_env( + tmp_path / "home", 45999, BETTER_MEMORY_STORAGE_BACKEND=None + ) + assert "BETTER_MEMORY_STORAGE_BACKEND" not in env + + +class TestAwsLockdownRoundTrip: + """A real CLI subprocess against the fake — the billing tripwire.""" + + def test_status_roundtrip_hits_only_local_fake_with_fake_sigv4( + self, clean_slate_home: Path + ) -> None: + bm_home = clean_slate_home / ".better-memory" + with FakeAgentCore() as fake: + write_fake_agentcore_json(bm_home) + + def _get_memory(request: RecordedRequest) -> dict: + # Path: /memories/{memoryId}/details — echo the id back. + memory_id = request.path.split("?", 1)[0].split("/")[2] + return { + "memory": { + "id": memory_id, + "arn": f"arn:aws:fake:::memory/{memory_id}", + "name": f"bm_e2e_{memory_id}", + "status": "ACTIVE", + "eventExpiryDuration": 90, + "strategies": [ + { + "strategyId": "STRAT-1", + "name": "bm_e2e_strategy", + "status": "ACTIVE", + } + ], + } + } + + fake.set_response("GetMemory", _get_memory) + env = agentcore_env(clean_slate_home, fake.port) + proc = subprocess.run( # noqa: S603 — test harness, fixed argv + [ + sys.executable, + "-m", + "better_memory.cli.main", + "agentcore", + "status", + "--home", + str(bm_home), + ], + env=env, + capture_output=True, + text=True, + timeout=60, + ) + + # Anti-vacuity: the round trip must be REAL — rc 0 with both + # memory ids reported ACTIVE, not an early error that never + # touched the wire. + assert proc.returncode == 0, proc.stderr + assert "EPI-FAKE-0001" in proc.stdout + assert "SEM-FAKE-0001" in proc.stdout + assert "ACTIVE" in proc.stdout + + requests = list(fake.requests) + assert len(requests) >= 2 + for request in requests: + # Zero requests escaped to *.amazonaws.com. + assert request.host.startswith("127.0.0.1:"), request.host + # SigV4 scope proves the FAKE credentials were signed with: + # the default chain never resolved real creds. + assert f"Credential={FAKE_ACCESS_KEY_ID}/" in request.authorization + assert request.operation == "GetMemory" + + +class TestDummyIdVarContainment: + """The FIXME(idvar-gate) workaround must never metastasize per-test.""" + + #: The ONLY files allowed to spell out the dummy-ID var names: + #: the single env helper, and the defect-pin negative test that gets + #: deleted together with the workaround when the product fix lands. + _ALLOWED = frozenset({"_agentcore_env.py", "test_agentcore_neg.py"}) + + @pytest.mark.parametrize("kind", ["SEMANTIC", "EPISODIC"]) + def test_dummy_idvar_literal_only_in_helper_and_defect_pin( + self, kind: str + ) -> None: + # Built by concatenation so THIS file never matches its own grep. + needle = "BETTER_MEMORY_AGENTCORE_" + kind + "_MEMORY_ID" + e2e_dir = Path(__file__).resolve().parent + + containing = { + path.name + for path in sorted(e2e_dir.glob("*.py")) + if needle in path.read_text(encoding="utf-8", errors="ignore") + } + + offenders = containing - self._ALLOWED + assert not offenders, ( + f"{needle} leaked outside the single helper + defect-pin test " + f"(see FIXME(idvar-gate) in _agentcore_env.py): {sorted(offenders)}. " + f"Use agentcore_env()/remove_dummy_id_vars_pins() instead." + ) + # Anti-vacuity: the two allowed files really do carry the literal — + # an over-eager cleanup that renamed the var would green this test + # while breaking the workaround. + assert self._ALLOWED <= containing, ( + f"expected {needle} in {sorted(self._ALLOWED)}, " + f"found only in {sorted(containing)}" + ) diff --git a/tests/e2e_meta/__init__.py b/tests/e2e_meta/__init__.py new file mode 100644 index 0000000..9449e12 --- /dev/null +++ b/tests/e2e_meta/__init__.py @@ -0,0 +1 @@ +"""Meta tests: prove the e2e harness itself is safe and meaningful.""" diff --git a/tests/e2e_meta/mutations/M1.patch b/tests/e2e_meta/mutations/M1.patch new file mode 100644 index 0000000..e322de5 --- /dev/null +++ b/tests/e2e_meta/mutations/M1.patch @@ -0,0 +1,13 @@ +diff --git a/better_memory/mcp/server.py b/better_memory/mcp/server.py +index 13dbaf7..2ac16d6 100644 +--- a/better_memory/mcp/server.py ++++ b/better_memory/mcp/server.py +@@ -162,7 +162,7 @@ def create_server() -> tuple[ + config = get_config() + + memory_conn = connect(config.memory_db) +- apply_migrations(memory_conn, migrations_dir=_MEMORY_MIGRATIONS) ++ # MUTATION M1: apply_migrations skipped for memory.db at boot. + knowledge_conn = connect(config.knowledge_db) + apply_migrations(knowledge_conn, migrations_dir=_KNOWLEDGE_MIGRATIONS) + diff --git a/tests/e2e_meta/mutations/M2.patch b/tests/e2e_meta/mutations/M2.patch new file mode 100644 index 0000000..830e2eb --- /dev/null +++ b/tests/e2e_meta/mutations/M2.patch @@ -0,0 +1,35 @@ +diff --git a/better_memory/hooks/session_bootstrap.py b/better_memory/hooks/session_bootstrap.py +index 52f75de..9baacac 100644 +--- a/better_memory/hooks/session_bootstrap.py ++++ b/better_memory/hooks/session_bootstrap.py +@@ -68,23 +68,19 @@ def main() -> None: + + try: + cfg = get_config() ++ # MUTATION M2: write_session_id hoisted above service.bootstrap() — ++ # the marker now lands even when bootstrap fails on a virgin home. ++ write_session_id( ++ cfg.home, ++ session_id, ++ project_dir=os.environ.get("CLAUDE_PROJECT_DIR") or cwd_str, ++ ) + with closing(connect(cfg.memory_db)) as conn: + service = SessionBootstrapService(conn) + result = service.bootstrap( + source=source, session_id=session_id, cwd=Path(cwd_str), + ) + rendered = result.additional_context +- # Bridge session_id to the MCP server: it doesn't see CLAUDE_SESSION_ID +- # in its spawn env. See better_memory/runtime/session_marker.py. +- # Resolution MUST match read_session_id's _resolve_project_dir order: +- # CLAUDE_PROJECT_DIR env first, then a non-empty fallback. The hook has +- # the payload's cwd as a stronger fallback than os.getcwd() because the +- # MCP server's cwd may not equal the project root. +- write_session_id( +- cfg.home, +- session_id, +- project_dir=os.environ.get("CLAUDE_PROJECT_DIR") or cwd_str, +- ) + except BaseException as exc: # noqa: BLE001 + try: + record_hook_error(hook_name="session_bootstrap", exc=exc) diff --git a/tests/e2e_meta/mutations/M3.patch b/tests/e2e_meta/mutations/M3.patch new file mode 100644 index 0000000..e2506b2 --- /dev/null +++ b/tests/e2e_meta/mutations/M3.patch @@ -0,0 +1,14 @@ +diff --git a/better_memory/hooks/contextual_inject.py b/better_memory/hooks/contextual_inject.py +index f158be2..fa580fd 100644 +--- a/better_memory/hooks/contextual_inject.py ++++ b/better_memory/hooks/contextual_inject.py +@@ -150,7 +150,8 @@ def main() -> None: + record_hook_error(hook_name="contextual_inject", exc=exc) + except BaseException: # noqa: BLE001 + pass +- rendered = "" ++ # MUTATION M3: except path exits WITHOUT printing the envelope. ++ sys.exit(0) + + try: + print( diff --git a/tests/e2e_meta/mutations/M4_seeded_breach.py b/tests/e2e_meta/mutations/M4_seeded_breach.py new file mode 100644 index 0000000..453f68d --- /dev/null +++ b/tests/e2e_meta/mutations/M4_seeded_breach.py @@ -0,0 +1,67 @@ +"""Mutation M4: hostile drop-in test — who watches the watchers. + +This is NOT a patch. ``scripts/e2e_mutation_smoke.py`` copies this file to +``tests/e2e/test_zz_seeded_breach.py`` inside a throwaway git worktree, arms +it with ``BM_E2E_SEEDED_BREACH=1``, and points the canary meta-run +(``tests/e2e_meta/test_canary_home.py``) at it via +``BM_E2E_CANARY_INNER_SCOPE``. The canary harness must then FAIL its +sentinel-hash / file-set-diff assertions (the isolation-breach messages +naming ``.claude.json`` and ``.better-memory/leak.txt``). If the harness +stays green with this file in the suite, the safety net is decorative — +that is the design §5 M4 verdict. + +Safety guards (belt and braces — this test must NEVER touch a real home): + +1. Skips unless ``BM_E2E_SEEDED_BREACH=1``, which only the mutation driver + sets. Someone running ``pytest tests/e2e`` in a worktree where the file + was accidentally left behind gets a skip, not a breach. +2. Skips unless ``Path.home()/.claude.json`` parses as JSON and carries the + ``__canary`` marker key seeded by + ``tests/e2e_meta/test_canary_home.py::_seed_canary_home`` — so even an + armed run refuses to write into anything but the seeded canary home. + +Note: this file deliberately does NOT use ``tests.e2e._env.isolated_env`` — +it runs *inside* the canary's inner pytest (whose HOME already points at the +canary) and its entire purpose is to escape per-test isolation the way a +buggy fixture would. +""" + +from __future__ import annotations + +import json +import os +from pathlib import Path + +import pytest + + +def test_zz_seeded_breach_writes_into_home() -> None: + """Deliberately corrupt the (canary) home so the harness must catch it.""" + if os.environ.get("BM_E2E_SEEDED_BREACH") != "1": + pytest.skip("seeded breach not armed (BM_E2E_SEEDED_BREACH != 1)") + + home = Path.home() + claude_json = home / ".claude.json" + if not claude_json.is_file(): + pytest.skip("no ~/.claude.json — refusing to breach a non-canary home") + try: + data = json.loads(claude_json.read_text(encoding="utf-8")) + except (json.JSONDecodeError, OSError): + pytest.skip("~/.claude.json unreadable — refusing to breach") + if not isinstance(data, dict) or "__canary" not in data: + pytest.skip("home is not the seeded canary — refusing to breach a real home") + + # Breach 1: mutate a sha256-pinned sentinel file + # (-> test_sentinel_files_byte_identical must fail naming .claude.json). + data["__breach"] = "M4 seeded breach" + claude_json.write_text(json.dumps(data, indent=2), encoding="utf-8") + + # Breach 2: drop a leak artifact under the canary's .better-memory tree + # (-> test_file_set_diff_is_empty must fail on the created path). + leak_dir = home / ".better-memory" + leak_dir.mkdir(exist_ok=True) + (leak_dir / "leak.txt").write_text("M4 seeded breach\n", encoding="utf-8") + + # The breach test itself PASSES — the canary harness, not this test, + # must be the thing that goes red. + assert (leak_dir / "leak.txt").is_file() diff --git a/tests/e2e_meta/test_canary_home.py b/tests/e2e_meta/test_canary_home.py new file mode 100644 index 0000000..5c0dd27 --- /dev/null +++ b/tests/e2e_meta/test_canary_home.py @@ -0,0 +1,269 @@ +"""Meta-run: the e2e suite executed under a seeded hostile canary home +(design F: ``meta-canary-home-run``). + +The single worst harness hazard, verified in source: ``install_hooks`` has +no ``--target`` flag — it writes ``Path.home()/.claude.json`` (line 339), +``~/.claude/settings.json`` and rmtree's skill dirs unconditionally. Any e2e +fixture that forgets to override USERPROFILE (Windows) or HOME (POSIX) would +therefore corrupt the real user's Claude config. This meta-test reruns the +suite in a subprocess whose HOME/USERPROFILE point at a seeded canary home +and proves, from outside, that nothing in the canary changed. + +Judge fixes applied: + +* The outer env starts from ``os.environ`` and **case-insensitively strips** + every ``BETTER_MEMORY_*`` / ``CLAUDE_*`` / ``AWS_*`` / ``OLLAMA_*`` key + (a dev shell exporting BETTER_MEMORY_HOME would otherwise shield an + un-redirected fixture from the canary), then overrides + HOME/USERPROFILE/HOMEDRIVE/HOMEPATH. ``BETTER_MEMORY_HOME`` is deliberately + NOT set: a correctly isolated suite must not need outer-shell protection, + so an un-redirected fixture falls back into ``/.better-memory`` + where it is caught. +* Explicit exit-5 vacuity assertion: suite deletion is reported as + "meta-run is vacuous", not as a generic failure. + +Inner-suite scoping contract (documented per the implementation task): + +* ``BM_E2E_CANARY_INNER_SCOPE`` unset and ``CI`` unset → REDUCED default + slice (fast local runs): ``test_install_hooks.py`` (the highest + real-home-corruption-risk surface) + ``test_hooks_contracts.py`` (sync + hook + MCP server spawns). +* ``BM_E2E_CANARY_INNER_SCOPE`` unset and ``CI`` set (every mainstream CI + exports it) → the FULL ``tests/e2e`` suite — the actual isolation proof. +* ``BM_E2E_CANARY_INNER_SCOPE=full`` → full ``tests/e2e`` anywhere. +* ``BM_E2E_CANARY_INNER_SCOPE=" [ ...]"`` → explicit targets. + +Sentinel assertions are decoupled from the inner exit code: each facet is +its own test against a module-scoped single run, so a failing inner suite +still gets its canary-integrity verdict reported separately. + +The inner run's own ``real_home_canary`` fixture plants dot-canary files +(``.bm-e2e-canary-*``) inside the canary home (its ``Path.home()``); they +are self-deleting but excluded from the file-set diff for robustness +against an interrupted inner run. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import subprocess +import sys +import uuid +from dataclasses import dataclass +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parents[2] + +SCOPE_VAR = "BM_E2E_CANARY_INNER_SCOPE" +REDUCED_DEFAULT: tuple[str, ...] = ( + "tests/e2e/test_install_hooks.py", + "tests/e2e/test_hooks_contracts.py", +) +STRIP_PREFIXES: tuple[str, ...] = ("BETTER_MEMORY_", "CLAUDE_", "AWS_", "OLLAMA_") + +#: The inner fixture's own additive artifacts — excluded from the diff. +CANARY_FIXTURE_PREFIX = ".bm-e2e-canary" + +#: Relative paths of every seeded sentinel file (sha256-pinned). +SENTINEL_FILES: tuple[str, ...] = ( + ".claude.json", + ".claude/settings.json", + ".claude/skills/user-skill/SKILL.md", + ".better-memory/memory.db", +) + +INNER_TIMEOUT = 540 # full tests/e2e comfortably fits; reduced slice ≪ this + + +def _set_ci(env: dict[str, str], key: str, value: str) -> None: + """Set ``key``, removing case-insensitive duplicates first (Windows).""" + for existing in [k for k in env if k.upper() == key.upper()]: + del env[existing] + env[key] = value + + +def _hostile_outer_env(canary: Path) -> dict[str, str]: + env = {k: v for k, v in os.environ.items() if not k.upper().startswith(STRIP_PREFIXES)} + # PYTEST_ADDOPTS could inject -m filters/plugins into the inner run and + # silently turn the meta-run vacuous — drop it. + for key in [k for k in env if k.upper() == "PYTEST_ADDOPTS"]: + del env[key] + _set_ci(env, "HOME", str(canary)) + _set_ci(env, "USERPROFILE", str(canary)) + if sys.platform == "win32": + drive, tail = os.path.splitdrive(str(canary)) + if drive: + _set_ci(env, "HOMEDRIVE", drive) + _set_ci(env, "HOMEPATH", tail or "\\") + return env + + +def _inner_scope() -> list[str]: + raw = os.environ.get(SCOPE_VAR, "").strip() + if raw == "full": + return ["tests/e2e"] + if raw: + return raw.split() + if os.environ.get("CI"): + return ["tests/e2e"] + return list(REDUCED_DEFAULT) + + +def _sha_map(canary: Path) -> dict[str, str]: + return { + rel: hashlib.sha256((canary / rel).read_bytes()).hexdigest() for rel in SENTINEL_FILES + } + + +def _path_set(canary: Path) -> set[str]: + """Full relative path set (files AND dirs), inner-fixture canaries excluded.""" + return { + p.relative_to(canary).as_posix() + for p in canary.rglob("*") + if not p.name.startswith(CANARY_FIXTURE_PREFIX) + } + + +def _seed_canary_home(canary: Path) -> str: + """Seed the hostile canary tree; returns the .claude.json uuid marker.""" + marker = str(uuid.uuid4()) + (canary / ".claude" / "skills" / "user-skill").mkdir(parents=True) + (canary / ".better-memory").mkdir() + (canary / ".claude.json").write_text( + json.dumps( + {"mcpServers": {"user-precious": {"command": "sentinel"}}, "__canary": marker}, + indent=2, + ), + encoding="utf-8", + ) + foreign_hook = {"type": "command", "command": "user-own-hook"} + (canary / ".claude" / "settings.json").write_text( + json.dumps( + {"hooks": {"SessionStart": [{"hooks": [foreign_hook]}]}}, + indent=2, + ), + encoding="utf-8", + ) + (canary / ".claude" / "skills" / "user-skill" / "SKILL.md").write_text( + f"user skill sentinel {uuid.uuid4()}\n", encoding="utf-8" + ) + # Deliberately NOT a valid sqlite file: any inner code path that opens + # the canary's memory.db errors loudly instead of silently mutating it. + (canary / ".better-memory" / "memory.db").write_bytes(os.urandom(64)) + return marker + + +@dataclass(frozen=True) +class CanaryRun: + canary: Path + scope: list[str] + rc: int + stdout: str + stderr: str + marker: str + pre_sha: dict[str, str] + pre_paths: set[str] + + +@pytest.fixture(scope="module") +def canary_run(tmp_path_factory: pytest.TempPathFactory) -> CanaryRun: + canary = tmp_path_factory.mktemp("canary-home") + marker = _seed_canary_home(canary) + pre_sha = _sha_map(canary) + pre_paths = _path_set(canary) + scope = _inner_scope() + proc = subprocess.run( + [ + sys.executable, + "-m", + "pytest", + "-q", + "--tb=short", + "-p", + "no:cacheprovider", + *scope, + ], + cwd=str(REPO_ROOT), + env=_hostile_outer_env(canary), + capture_output=True, + text=True, + timeout=INNER_TIMEOUT, + ) + return CanaryRun( + canary=canary, + scope=scope, + rc=proc.returncode, + stdout=proc.stdout, + stderr=proc.stderr, + marker=marker, + pre_sha=pre_sha, + pre_paths=pre_paths, + ) + + +def _tail(text: str, lines: int = 40) -> str: + return "\n".join(text.splitlines()[-lines:]) + + +class TestInnerRun: + def test_inner_run_not_vacuous(self, canary_run: CanaryRun) -> None: + """Exit-5 means pytest collected NOTHING — a deleted/renamed suite + must be reported as vacuity, not mistaken for a green isolation proof + (judge fix).""" + assert canary_run.rc != 5, ( + f"{canary_run.scope} collected nothing — meta-run is vacuous\n" + f"{_tail(canary_run.stdout)}" + ) + + def test_inner_suite_green_under_hostile_home(self, canary_run: CanaryRun) -> None: + """The suite passes with HOME pointed at a hostile location — proves + no test depends on (or requires protection of) the real home.""" + assert canary_run.rc == 0, ( + f"inner pytest rc={canary_run.rc} for scope {canary_run.scope}\n" + f"--- stdout tail ---\n{_tail(canary_run.stdout)}\n" + f"--- stderr tail ---\n{_tail(canary_run.stderr)}" + ) + assert "passed" in canary_run.stdout, _tail(canary_run.stdout) + + +class TestCanaryIntegrity: + """Sentinel facets — asserted regardless of the inner exit code.""" + + def test_sentinel_files_byte_identical(self, canary_run: CanaryRun) -> None: + post_sha = {} + for rel in SENTINEL_FILES: + path = canary_run.canary / rel + assert path.exists(), f"seeded sentinel vanished: {rel}" + post_sha[rel] = hashlib.sha256(path.read_bytes()).hexdigest() + changed = [rel for rel in SENTINEL_FILES if post_sha[rel] != canary_run.pre_sha[rel]] + assert not changed, ( + f"HARNESS ISOLATION BREACH: canary files mutated by the inner run: {changed}" + ) + + def test_file_set_diff_is_empty(self, canary_run: CanaryRun) -> None: + """No leak artifacts appeared: no /.better-memory/ + install-backups or spool, no better-memory-* skill entries, nothing + deleted.""" + post_paths = _path_set(canary_run.canary) + created = sorted(post_paths - canary_run.pre_paths) + deleted = sorted(canary_run.pre_paths - post_paths) + assert created == [], f"inner run CREATED paths in the canary home: {created}" + assert deleted == [], f"inner run DELETED paths from the canary home: {deleted}" + + def test_mcp_servers_semantics_preserved(self, canary_run: CanaryRun) -> None: + """Parsed-JSON subset check: catches install_hooks leaking through a + forgotten USERPROFILE even if serialization were byte-stable.""" + data = json.loads((canary_run.canary / ".claude.json").read_text(encoding="utf-8")) + assert data["mcpServers"] == {"user-precious": {"command": "sentinel"}} + assert "better-memory" not in data["mcpServers"] + assert data["__canary"] == canary_run.marker + + def test_user_skill_dir_survives(self, canary_run: CanaryRun) -> None: + """Catches install_skill_symlinks' backup-less rmtree escaping + isolation.""" + skill = canary_run.canary / ".claude" / "skills" / "user-skill" / "SKILL.md" + assert skill.is_file() diff --git a/tests/e2e_meta/test_env_bleed.py b/tests/e2e_meta/test_env_bleed.py new file mode 100644 index 0000000..6f10191 --- /dev/null +++ b/tests/e2e_meta/test_env_bleed.py @@ -0,0 +1,188 @@ +"""Poisoned-shell re-run of a T1 slice (design F: ``meta-env-bleed-poisoned-shell``). + +Dev-shell ``BETTER_MEMORY_*`` / ``AWS_*`` / ``OLLAMA_*`` / ``CLAUDE_*`` +pollution must not change e2e outcomes. The failure mechanism is real and +verified in source: ``config.py:293-301`` raises ValueError pre-handshake +when ``BETTER_MEMORY_STORAGE_BACKEND=agentcore`` leaks without the ID vars, +so any fixture that reintroduces the ``{**os.environ, ...}`` spawn pattern +(the historical tests/mcp precedent) dies loudly under this poison; a +fixture forgetting to pin ``BETTER_MEMORY_PROJECT`` surfaces as +``poison-project`` scoping failures. + +Mechanism: build a maximally hostile outer env ON TOP of the real +``os.environ`` (poison is added, nothing is stripped — that is the point), +then re-run the sqlite-mode T1 slice whose assertions the poisoned +``backend=agentcore`` / ``embeddings=ollama`` values would break if any +fixture inherited them. A decoy ``BETTER_MEMORY_HOME`` dir with a hashed +sentinel pins location bleed independently of pass/fail outcomes. + +Deviation from the author scenario (documented): HOME/USERPROFILE are +redirected to a neutral tmp home in the hostile env. The scenario's +assertions never involve the real home, and leaving the real HOME in place +would make the inner run's session-scoped ``real_home_canary`` fixture +plant/delete the SAME-named dot-canary files in the real home as an outer +``pytest tests`` session's fixture (both resolve worker id 'main'), +producing a false HARNESS ISOLATION BREACH at outer teardown. The neutral +home removes that collision without weakening any poison assertion. +""" + +from __future__ import annotations + +import hashlib +import os +import subprocess +import sys +import uuid +from dataclasses import dataclass +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parents[2] + +#: The T1 sqlite happy-path + hook-contract modules: they assert sqlite-mode +#: behavior that an inherited backend=agentcore / embeddings=ollama poison +#: would break, making them the highest-signal bleed detectors. +INNER_SLICE: tuple[str, ...] = ( + "tests/e2e/test_sqlite_journey.py", + "tests/e2e/test_hooks_contracts.py", +) + +#: Canonical-cased poison values (fixed strings; ``.invalid`` is +#: RFC-2606-reserved so DNS can never resolve; us-gov-west-1 and +#: AWS_PROFILE=poison point at nothing real). +POISON_VARS: dict[str, str] = { + "BETTER_MEMORY_STORAGE_BACKEND": "agentcore", + "BETTER_MEMORY_AGENTCORE_SEMANTIC_MEMORY_ID": "POISON-SEM", + "BETTER_MEMORY_AGENTCORE_EPISODIC_MEMORY_ID": "POISON-EPI", + "BETTER_MEMORY_AGENTCORE_REGION": "us-gov-west-1", + "BETTER_MEMORY_PROJECT": "poison-project", + "BETTER_MEMORY_CONTEXT_INJECT_MODE": "aggressive", + "BETTER_MEMORY_EMBEDDINGS_BACKEND": "ollama", + "OLLAMA_HOST": "http://does-not-exist.invalid:1", + "CLAUDE_SESSION_ID": "poison-session", + "AWS_PROFILE": "poison", + "AWS_DEFAULT_REGION": "us-east-1", +} + +#: Poison markers that must never appear in inner output — catches partial +#: bleed where a var leaks into one subprocess's error message even if the +#: tests still pass. +POISON_MARKERS: tuple[str, ...] = ("poison-project", "POISON-SEM") + +INNER_TIMEOUT = 420 + + +def _set_ci(env: dict[str, str], key: str, value: str) -> None: + """Set ``key``, removing case-insensitive duplicates first (Windows env + keys carry arbitrary case — a naive assignment can leave 'Ollama_Host' + alive next to 'OLLAMA_HOST').""" + for existing in [k for k in env if k.upper() == key.upper()]: + del env[existing] + env[key] = value + + +def _poisoned_env(decoy_home: Path, neutral_home: Path) -> dict[str, str]: + env = dict(os.environ) # deliberately inherit-everything: worst case + for key, value in POISON_VARS.items(): + _set_ci(env, key, value) + _set_ci(env, "BETTER_MEMORY_HOME", str(decoy_home)) + _set_ci(env, "HOME", str(neutral_home)) + _set_ci(env, "USERPROFILE", str(neutral_home)) + if sys.platform == "win32": + drive, tail = os.path.splitdrive(str(neutral_home)) + if drive: + _set_ci(env, "HOMEDRIVE", drive) + _set_ci(env, "HOMEPATH", tail or "\\") + # PYTEST_ADDOPTS could inject -m filters into the inner run and turn the + # bleed check vacuous. + for key in [k for k in env if k.upper() == "PYTEST_ADDOPTS"]: + del env[key] + return env + + +@dataclass(frozen=True) +class BleedRun: + rc: int + stdout: str + stderr: str + decoy: Path + sentinel: Path + sentinel_sha: str + + +@pytest.fixture(scope="module") +def bleed_run(tmp_path_factory: pytest.TempPathFactory) -> BleedRun: + decoy = tmp_path_factory.mktemp("decoy-bm-home") + sentinel = decoy / "sentinel.txt" + sentinel.write_text(f"decoy sentinel {uuid.uuid4()}\n", encoding="utf-8") + sentinel_sha = hashlib.sha256(sentinel.read_bytes()).hexdigest() + neutral_home = tmp_path_factory.mktemp("neutral-home") + + proc = subprocess.run( + [ + sys.executable, + "-m", + "pytest", + "-q", + "--tb=short", + "-p", + "no:cacheprovider", + *INNER_SLICE, + ], + cwd=str(REPO_ROOT), + env=_poisoned_env(decoy, neutral_home), + capture_output=True, + text=True, + timeout=INNER_TIMEOUT, + ) + return BleedRun( + rc=proc.returncode, + stdout=proc.stdout, + stderr=proc.stderr, + decoy=decoy, + sentinel=sentinel, + sentinel_sha=sentinel_sha, + ) + + +def _tail(text: str, lines: int = 40) -> str: + return "\n".join(text.splitlines()[-lines:]) + + +def test_inner_slice_not_vacuous(bleed_run: BleedRun) -> None: + assert bleed_run.rc != 5, ( + f"{list(INNER_SLICE)} collected nothing — bleed check is vacuous\n" + f"{_tail(bleed_run.stdout)}" + ) + + +def test_inner_slice_green_under_poisoned_shell(bleed_run: BleedRun) -> None: + """sqlite-mode tests still pass with backend=agentcore + POISON ids + + aggressive inject mode + embeddings=ollama in the outer shell — proving + every fixture constructs its env from the explicit allowlist rather than + inheriting.""" + assert bleed_run.rc == 0, ( + f"inner pytest rc={bleed_run.rc}\n" + f"--- stdout tail ---\n{_tail(bleed_run.stdout)}\n" + f"--- stderr tail ---\n{_tail(bleed_run.stderr)}" + ) + assert "passed" in bleed_run.stdout, _tail(bleed_run.stdout) + + +def test_decoy_home_untouched(bleed_run: BleedRun) -> None: + """No test used the outer BETTER_MEMORY_HOME's location: the decoy still + contains exactly its one sentinel file with an unchanged hash.""" + files = [p for p in bleed_run.decoy.rglob("*")] + assert files == [bleed_run.sentinel], ( + f"decoy BETTER_MEMORY_HOME was touched: {[str(p) for p in files]}" + ) + assert hashlib.sha256(bleed_run.sentinel.read_bytes()).hexdigest() == bleed_run.sentinel_sha + + +def test_no_poison_strings_in_inner_output(bleed_run: BleedRun) -> None: + combined = bleed_run.stdout + bleed_run.stderr + leaked = [marker for marker in POISON_MARKERS if marker in combined] + assert leaked == [], ( + f"poison values bled into inner-run output: {leaked}\n{_tail(combined, 60)}" + ) diff --git a/tests/e2e_meta/test_env_helper_contract.py b/tests/e2e_meta/test_env_helper_contract.py new file mode 100644 index 0000000..385d3f0 --- /dev/null +++ b/tests/e2e_meta/test_env_helper_contract.py @@ -0,0 +1,492 @@ +"""Contract tests for the e2e harness env choke point (design F: meta-env-helper-contract). + +Four lenses from the harness-safety scenario +``harness-windows-posix-env-helper-contract``: + +* **A — dict-level contract**: ``isolated_env`` sets HOME+USERPROFILE on every + OS, preserves only the system allowlist, strips all + CLAUDE_*/BETTER_MEMORY_*/AWS_*/OLLAMA_* case-insensitively, and yields + case-insensitively-unique keys. +* **B — child-process ground truth**: a real spawned python reports + ``Path.home()``/``expanduser('~')`` == tmp and can import ssl/sqlite3 + (SYSTEMROOT preserved) — cannot be fooled by dict-level mocking. +* **C — single-choke-point enforcement**: every spawn site in ``tests/e2e`` + derives its env from the helper; hand-rolled env dicts and + ``os.environ``-copy patterns are structurally banned. +* **D — skip-decoration checks**: setup.sh tests skip (not error) without + bash; symlink content assertions are capability-probed. + +Plus the ``mcp_session`` boot smoke: proves the shared helper can drive the +real MCP server hermetically before any journey test builds on it. +""" + +from __future__ import annotations + +import ast +import io +import json +import os +import re +import subprocess +import sys +from pathlib import Path +from typing import Any + +import pytest + +from tests.e2e._env import ( + ALLOWLIST, + POISONED_OLLAMA_HOST, + STRIPPED_PREFIXES, + isolated_env, +) +from tests.e2e.conftest import mcp_session, run_hook + +E2E_DIR = Path(__file__).resolve().parents[1] / "e2e" + +#: The only STRIPPED_PREFIXES-matching keys isolated_env may emit by default — +#: each one a deliberate pin, not an inherited value. +DELIBERATE_PINS = { + "BETTER_MEMORY_HOME", + "BETTER_MEMORY_PROJECT", + "BETTER_MEMORY_EMBEDDINGS_BACKEND", + "CLAUDE_SESSION_ID", + "OLLAMA_HOST", +} + + +# --------------------------------------------------------------------------- +# Contract test A — dict-level +# --------------------------------------------------------------------------- + + +class TestContractADictLevel: + def test_home_and_userprofile_both_set_on_every_os(self, tmp_path: Path) -> None: + """Both vars, unconditionally — the POSIX-contributor-deletes- + USERPROFILE decay path is the exact bug that torches a Windows dev's + real ~/.claude.json.""" + env = isolated_env(tmp_path) + assert env["HOME"] == str(tmp_path) + assert env["USERPROFILE"] == str(tmp_path) + if sys.platform == "win32": + drive, tail = os.path.splitdrive(str(tmp_path)) + assert env["HOMEDRIVE"] == drive + assert env["HOMEPATH"] == tail + assert env["HOMEDRIVE"] + env["HOMEPATH"] == str(tmp_path) + + def test_hostile_outer_vars_stripped_case_insensitively( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.setenv("BETTER_MEMORY_STORAGE_BACKEND", "agentcore") + monkeypatch.setenv("AWS_PROFILE", "prod") + monkeypatch.setenv("CLAUDE_PROJECT_DIR", "C:/real") + # Mixed/lower case — Windows env keys carry arbitrary case and a + # case-sensitive strip would let these through on POSIX CI. + monkeypatch.setenv("aws_secret_access_key", "AKIA-POISON") + monkeypatch.setenv("Ollama_Host", "http://real-ollama:11434") + + env = isolated_env(tmp_path) + uppers = {k.upper() for k in env} + assert "BETTER_MEMORY_STORAGE_BACKEND" not in uppers + assert "AWS_PROFILE" not in uppers + assert "CLAUDE_PROJECT_DIR" not in uppers + assert "AWS_SECRET_ACCESS_KEY" not in uppers + assert "AKIA-POISON" not in env.values() + # The poison pin won over the outer daemon address. + assert env["OLLAMA_HOST"] == POISONED_OLLAMA_HOST + assert env["OLLAMA_HOST"].endswith(".invalid:1") + + def test_only_deliberate_pins_carry_stripped_prefixes( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + """Exact-set assertion: any new prefixed key appearing by default is + either an inherited leak or an undocumented pin — both must be + deliberate, so this test must be updated consciously.""" + monkeypatch.setenv("CLAUDE_CODE_SESSION_ID", "leak") + monkeypatch.setenv("BETTER_MEMORY_CONTEXT_INJECT_MODE", "aggressive") + monkeypatch.setenv("AWS_ENDPOINT_URL", "https://bedrock.example") + + env = isolated_env(tmp_path) + prefixed = {k.upper() for k in env if k.upper().startswith(STRIPPED_PREFIXES)} + assert prefixed == DELIBERATE_PINS + + def test_system_critical_allowlist_preserved(self, tmp_path: Path) -> None: + env = isolated_env(tmp_path) + uppers = {k.upper() for k in env} + assert "PATH" in uppers + if sys.platform == "win32": + # Child pythons fail to import ssl/sqlite3 without SystemRoot; + # COMSPEC is required by anything that shells out. + assert "SYSTEMROOT" in uppers + assert "COMSPEC" in uppers + + def test_non_allowlisted_outer_var_not_inherited( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + """Allowlist-built, not strip-built: an arbitrary outer var (no + stripped prefix) still never reaches the child.""" + monkeypatch.setenv("E2E_RANDOM_SENTINEL", "leak-me") + env = isolated_env(tmp_path) + assert "E2E_RANDOM_SENTINEL" not in {k.upper() for k in env} + assert "leak-me" not in env.values() + + def test_case_insensitive_key_uniqueness( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + """Duplicate 'Path'/'PATH' style keys corrupt Windows CreateProcess + env blocks; the helper must dedupe by upper-cased name.""" + if sys.platform != "win32": + # POSIX os.environ CAN hold case-colliding keys — seed a pair. + monkeypatch.setenv("Temp", "/case/one") + monkeypatch.setenv("TEMP", "/case/two") + env = isolated_env(tmp_path) + uppers = [k.upper() for k in env] + assert len(set(uppers)) == len(uppers), sorted(env) + + def test_pins_override_none_removes_and_extras_added(self, tmp_path: Path) -> None: + env = isolated_env( + tmp_path, + BETTER_MEMORY_PROJECT="custom-proj", + CLAUDE_SESSION_ID=None, + EXTRA_PIN="extra-value", + ) + assert env["BETTER_MEMORY_PROJECT"] == "custom-proj" + assert "CLAUDE_SESSION_ID" not in {k.upper() for k in env} + assert env["EXTRA_PIN"] == "extra-value" + + def test_default_pin_values(self, tmp_path: Path) -> None: + env = isolated_env(tmp_path) + assert env["BETTER_MEMORY_HOME"] == str(tmp_path / ".better-memory") + assert env["BETTER_MEMORY_EMBEDDINGS_BACKEND"] == "sqlite" + assert env["OLLAMA_HOST"] == POISONED_OLLAMA_HOST + assert env["CLAUDE_SESSION_ID"] == "e2e-session-1" + assert env["BETTER_MEMORY_PROJECT"] == "e2e-project" + + def test_allowlist_and_stripped_prefixes_are_disjoint(self) -> None: + """A prefixed name sneaking into the allowlist would reopen the leak.""" + assert not {n for n in ALLOWLIST if n.startswith(STRIPPED_PREFIXES)} + + +# --------------------------------------------------------------------------- +# Contract test B — child-process ground truth +# --------------------------------------------------------------------------- + +_CHILD_PROBE = ( + "from pathlib import Path\n" + "import os, json, tempfile, sqlite3, ssl\n" # ssl/sqlite3: SYSTEMROOT proof on Windows + "print(json.dumps({\n" + " 'home': str(Path.home()),\n" + " 'expanduser': os.path.expanduser('~'),\n" + " 'gettempdir': tempfile.gettempdir(),\n" + " 'prefixed': sorted(\n" + " k.upper() for k in os.environ\n" + " if k.upper().startswith(('CLAUDE_', 'BETTER_MEMORY_', 'AWS_', 'OLLAMA_'))\n" + " ),\n" + "}))\n" +) + + +class TestContractBChildGroundTruth: + def _spawn_probe(self, env: dict[str, str]) -> dict[str, Any]: + proc = subprocess.run( + [sys.executable, "-c", _CHILD_PROBE], + env=env, + capture_output=True, + text=True, + timeout=30, + ) + assert proc.returncode == 0, f"child probe died: {proc.stderr}" + return json.loads(proc.stdout.strip().splitlines()[-1]) + + def test_child_home_and_expanduser_land_in_tmp(self, tmp_path: Path) -> None: + report = self._spawn_probe(isolated_env(tmp_path)) + assert report["home"] == str(tmp_path) + assert report["expanduser"] == str(tmp_path) + # TEMP/TMP preserved: gettempdir resolved without crashing or + # falling back to the cwd. + assert report["gettempdir"] + + def test_child_sees_only_deliberate_prefixed_pins( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + """Hostile outer shell → the child's actual environment (not just the + dict we built) contains exactly the deliberate pins.""" + monkeypatch.setenv("AWS_ACCESS_KEY_ID", "AKIA-REAL") + monkeypatch.setenv("CLAUDE_PROJECT_DIR", str(Path.home())) + monkeypatch.setenv("BETTER_MEMORY_STORAGE_BACKEND", "agentcore") + report = self._spawn_probe(isolated_env(tmp_path)) + assert set(report["prefixed"]) == DELIBERATE_PINS + + +# --------------------------------------------------------------------------- +# Contract test C — single-choke-point enforcement (lint-as-test) +# --------------------------------------------------------------------------- + +_HELPER_NAME = re.compile(r"\b(isolated_env|agentcore_env)\b") +_SUBPROCESS_FUNC = re.compile(r"^(subprocess\.)?(run|Popen|check_output|check_call|call)$") +#: Parameter names allowed to carry a caller-supplied env (the conftest +#: wrapper helpers). Anything else must trace back to a helper call. +_ENV_PARAM = re.compile(r"^(env|.*_env)$") + + +def _func_source(src: str, node: ast.Call) -> str: + return (ast.get_source_segment(src, node.func) or "").strip() + + +def _blessed_names(src: str, tree: ast.AST) -> set[str]: + """Names whose value provably derives from isolated_env/agentcore_env. + + Seeds: function parameters named ``env``/``*_env`` (their values flow + from call sites, which are themselves checked). Fixed point over simple + assignments whose RHS mentions a helper or an already-blessed name. + """ + blessed: set[str] = set() + assignments: list[tuple[list[str], str]] = [] + for node in ast.walk(tree): + if isinstance(node, ast.FunctionDef | ast.AsyncFunctionDef): + a = node.args + for arg in [*a.posonlyargs, *a.args, *a.kwonlyargs]: + if _ENV_PARAM.match(arg.arg): + blessed.add(arg.arg) + elif isinstance(node, ast.Assign): + names = [t.id for t in node.targets if isinstance(t, ast.Name)] + if names: + assignments.append((names, ast.get_source_segment(src, node.value) or "")) + elif isinstance(node, ast.AnnAssign) and node.value is not None: + if isinstance(node.target, ast.Name): + assignments.append( + ([node.target.id], ast.get_source_segment(src, node.value) or "") + ) + changed = True + while changed: + changed = False + for names, rhs in assignments: + if all(n in blessed for n in names): + continue + ok = bool(_HELPER_NAME.search(rhs)) or any( + re.search(rf"\b{re.escape(b)}\b", rhs) for b in blessed + ) + if ok: + for n in names: + if n not in blessed: + blessed.add(n) + changed = True + return blessed + + +def _spawn_env_violations(path: Path) -> list[str]: + src = path.read_text(encoding="utf-8") + tree = ast.parse(src, filename=str(path)) + blessed = _blessed_names(src, tree) + violations: list[str] = [] + + def env_ok(expr: ast.expr | None, where: str) -> None: + if expr is None: + violations.append(f"{path.name}:{where}: spawn without explicit env=") + return + segment = ast.get_source_segment(src, expr) or "" + if _HELPER_NAME.search(segment): + return + if isinstance(expr, ast.Name) and expr.id in blessed: + return + violations.append( + f"{path.name}:{where}: env not derived from isolated_env/agentcore_env: {segment!r}" + ) + + for node in ast.walk(tree): + if not isinstance(node, ast.Call): + continue + func = _func_source(src, node) + kw_env = next((k.value for k in node.keywords if k.arg == "env"), None) + line = f"line {node.lineno}" + if _SUBPROCESS_FUNC.match(func) or func.endswith("StdioServerParameters"): + env_ok(kw_env, line) + elif func.endswith("run_hook"): + pos = node.args[2] if len(node.args) >= 3 else None + env_ok(kw_env if kw_env is not None else pos, line) + elif func.endswith("mcp_session"): + pos = node.args[0] if node.args else None + env_ok(kw_env if kw_env is not None else pos, line) + return violations + + +def _environ_copy_violations(path: Path) -> list[str]: + """Ban os.environ.copy(), dict(os.environ) and {**os.environ, ...}.""" + src = path.read_text(encoding="utf-8") + tree = ast.parse(src, filename=str(path)) + violations: list[str] = [] + for node in ast.walk(tree): + if isinstance(node, ast.Call): + func = _func_source(src, node) + if func.endswith("environ.copy"): + violations.append(f"{path.name}:{node.lineno}: os.environ.copy()") + elif func == "dict" and any( + "environ" in (ast.get_source_segment(src, a) or "") for a in node.args + ): + violations.append(f"{path.name}:{node.lineno}: dict(os.environ)") + elif isinstance(node, ast.Dict): + for key, value in zip(node.keys, node.values, strict=True): + if key is None and "environ" in (ast.get_source_segment(src, value) or ""): + violations.append(f"{path.name}:{node.lineno}: {{**os.environ, ...}}") + return violations + + +def _e2e_python_files() -> list[Path]: + return sorted(p for p in E2E_DIR.rglob("*.py") if "__pycache__" not in p.parts) + + +class TestContractCSingleChokePoint: + def test_e2e_dir_exists_and_scanned(self) -> None: + """Anti-vacuity: the scan target exists and includes the helpers.""" + files = {p.name for p in _e2e_python_files()} + assert "_env.py" in files + assert "conftest.py" in files + + def test_every_spawn_site_uses_the_helper(self) -> None: + violations = [v for p in _e2e_python_files() for v in _spawn_env_violations(p)] + assert not violations, "\n".join(violations) + + def test_no_os_environ_copy_patterns(self) -> None: + violations = [v for p in _e2e_python_files() for v in _environ_copy_violations(p)] + assert not violations, "\n".join(violations) + + def test_checker_flags_hand_rolled_env_dict(self, tmp_path: Path) -> None: + """The lint itself must go red on the banned patterns (guard against + the checker decaying into a green-always ritual).""" + bad = tmp_path / "test_bad.py" + bad.write_text( + "import subprocess, sys, os\n" + "def test_x(tmp_path):\n" + " env = {'HOME': str(tmp_path)}\n" + " subprocess.run([sys.executable, '-V'], env=env)\n" + " subprocess.run([sys.executable, '-V'])\n" + " e2 = {**os.environ, 'HOME': str(tmp_path)}\n" + " subprocess.run([sys.executable, '-V'], env=os.environ.copy())\n", + encoding="utf-8", + ) + spawn = _spawn_env_violations(bad) + copies = _environ_copy_violations(bad) + assert len(spawn) == 3, spawn # hand-rolled name, missing env=, environ.copy expr + assert any("**os.environ" in v for v in copies) + assert any("environ.copy" in v for v in copies) + + def test_checker_accepts_helper_derived_env(self, tmp_path: Path) -> None: + good = tmp_path / "test_good.py" + good.write_text( + "import subprocess, sys\n" + "from tests.e2e._env import isolated_env\n" + "from tests.e2e.conftest import mcp_session, run_hook\n" + "def test_x(tmp_path):\n" + " env = isolated_env(tmp_path)\n" + " env2 = {**env, 'EXTRA': '1'}\n" + " subprocess.run([sys.executable, '-V'], env=env2)\n" + " run_hook('m', {}, env)\n" + " subprocess.run([sys.executable, '-V'], env=isolated_env(tmp_path))\n", + encoding="utf-8", + ) + assert _spawn_env_violations(good) == [] + assert _environ_copy_violations(good) == [] + + +# --------------------------------------------------------------------------- +# Contract test D — skip-decoration checks +# --------------------------------------------------------------------------- + + +class TestContractDSkipDecorations: + def test_setup_sh_module_skips_without_bash(self) -> None: + mod = E2E_DIR / "test_setup_sh.py" + if not mod.exists(): + pytest.skip("tests/e2e/test_setup_sh.py not implemented yet (design task 9)") + src = mod.read_text(encoding="utf-8") + assert "skipif" in src, "setup.sh module must SKIP (not error) when bash is absent" + assert re.search(r"which\(\s*['\"]bash['\"]\s*\)", src), ( + "expected a shutil.which('bash') presence probe driving the skipif" + ) + + def test_symlink_assertions_are_capability_probed(self) -> None: + """Modules asserting symlink outcomes must gate them behind a probe + (mirrors tests/cli/test_install_skill_symlink.py's requires_symlinks): + non-Developer-Mode Windows boxes must skip the symlink ASSERT while + still running the installer-exit-0 assertions.""" + probe = re.compile( + r"requires_symlinks|_symlinks_available|symlinks?_(available|supported)|can_symlink" + ) + offenders = [ + p.name + for p in _e2e_python_files() + if ".is_symlink()" in p.read_text(encoding="utf-8") + and not probe.search(p.read_text(encoding="utf-8")) + ] + assert not offenders, ( + f"symlink assertions without a capability probe: {offenders}" + ) + + +# --------------------------------------------------------------------------- +# Helper smoke: run_hook plumbing + mcp_session hermetic boot +# --------------------------------------------------------------------------- + + +def _single_json_dict(result: Any) -> dict[str, Any]: + content = result.content + assert not getattr(result, "isError", False), f"tool errored: {content!r}" + assert len(content) == 1, f"expected one content block: {content!r}" + block = content[0] + assert getattr(block, "type", None) == "text" + parsed = json.loads(block.text) + assert isinstance(parsed, dict) + return parsed + + +class TestHelperSmoke: + def test_run_hook_pipes_stdin_and_returns_triple(self, tmp_path: Path) -> None: + """Plumbing-only smoke via a stdlib module: payload reaches the child's + stdin, (rc, stdout, stderr) come back. Hook behavioral contracts are + owned by the journey/contract test tasks.""" + rc, out, err = run_hook("json.tool", {"probe": 1}, isolated_env(tmp_path)) + assert rc == 0, err + assert json.loads(out) == {"probe": 1} + + async def test_mcp_session_boots_real_server_hermetically(self, tmp_path: Path) -> None: + """The proven-helper smoke for every T1/T2 builder: the real MCP + server boots offline on a virgin fake home (sqlite embeddings, + poisoned OLLAMA_HOST), answers list_tools + memory.retrieve, and all + disk writes land under the fake home.""" + home = tmp_path / "home" + home.mkdir() + env = isolated_env(home) + errlog_path = tmp_path / "server.stderr" # outside the fake home + + with errlog_path.open("w", encoding="utf-8") as errlog: + async with mcp_session(env, errlog=errlog) as session: + listed = await session.list_tools() + names = {tool.name for tool in listed.tools} + assert {"memory.observe", "memory.retrieve", "knowledge.search"} <= names + + payload = _single_json_dict(await session.call_tool("memory.retrieve", {})) + assert isinstance(payload["do"], list) + assert isinstance(payload["dont"], list) + assert isinstance(payload["neutral"], list) + + # Hermetic ground truth: the server's DB landed under the FAKE home. + assert (home / ".better-memory" / "memory.db").exists(), sorted( + p.relative_to(tmp_path).as_posix() for p in tmp_path.rglob("*") + ) + + async def test_mcp_session_rejects_env_missing_userprofile(self, tmp_path: Path) -> None: + """Guard branch: the helper refuses an env without USERPROFILE/HOME + instead of letting the SDK leak the real profile underneath it.""" + env = isolated_env(tmp_path) + stripped = {k: v for k, v in env.items() if k.upper() != "USERPROFILE"} + with pytest.raises(ValueError, match="USERPROFILE"): + async with mcp_session(stripped): + pass # pragma: no cover — must not spawn + + async def test_mcp_session_rejects_fileless_errlog(self, tmp_path: Path) -> None: + """Guard branch: errlog must have a real fileno() — the SDK hands it + to subprocess creation as the stderr handle; StringIO silently loses + every server-side traceback.""" + with pytest.raises(io.UnsupportedOperation): + async with mcp_session(isolated_env(tmp_path), errlog=io.StringIO()): + pass # pragma: no cover — must not spawn diff --git a/tests/e2e_meta/test_marker_wiring.py b/tests/e2e_meta/test_marker_wiring.py new file mode 100644 index 0000000..c097223 --- /dev/null +++ b/tests/e2e_meta/test_marker_wiring.py @@ -0,0 +1,154 @@ +"""Marker / tier wiring (design F: meta-marker-tier-wiring). + +Pins the collection contract that keeps the default run hermetic: + +* plain ``pytest`` collects T1+T2 (tests/e2e) and ZERO nodes from the live + T3 module — the default run can never touch real AWS; +* the ``e2e`` marker is declared in pyproject (no unknown-mark warning); +* ``-m integration`` without ``BETTER_MEMORY_TEST_AGENTCORE=1`` goes + green-**skipped** — never red, never exit-5 no-tests-collected — matching + tests/integration/conftest.py's ``pytest.skip`` gate. + +Each check spawns a real pytest subprocess with an ``isolated_env``-built +environment, so a dev box with ``BETTER_MEMORY_TEST_AGENTCORE=1`` (or real +AWS creds) exported cannot turn the gate check into a live AWS run. +""" + +from __future__ import annotations + +import re +import subprocess +import sys +from pathlib import Path + +import pytest + +from tests.e2e._env import isolated_env + +REPO_ROOT = Path(__file__).resolve().parents[2] +E2E_DIR = REPO_ROOT / "tests" / "e2e" +#: The T3 live-AWS module (design task 12; may not exist yet mid-phase). +LIVE_T3 = REPO_ROOT / "tests" / "integration" / "test_agentcore_live_e2e.py" +#: Existing fully integration-marked module using the same skip gate — +#: fallback target until the T3 module lands. +ROUNDTRIP = REPO_ROOT / "tests" / "integration" / "test_agentcore_roundtrip.py" + + +def _run_pytest( + args: list[str], tmp_path: Path, *, timeout: float = 240 +) -> subprocess.CompletedProcess[str]: + home = tmp_path / "pytest-home" + home.mkdir(exist_ok=True) + # isolated_env strips BETTER_MEMORY_TEST_AGENTCORE(+_KEEP), all AWS_* + # creds and PYTEST_ADDOPTS — the spawned pytest sees only pyproject + # defaults and can never reach live AWS. + env = isolated_env(home) + return subprocess.run( + [sys.executable, "-m", "pytest", "-p", "no:cacheprovider", *args], + cwd=str(REPO_ROOT), + env=env, + capture_output=True, + text=True, + timeout=timeout, + ) + + +def _node_ids(stdout: str) -> list[str]: + return [line.strip() for line in stdout.splitlines() if "::" in line] + + +def _e2e_has_test_modules() -> bool: + return any(E2E_DIR.glob("test_*.py")) + + +def test_default_collection_excludes_live_t3(tmp_path: Path) -> None: + """Plain pytest (default addopts) on tests/e2e + tests/integration must + never pick up a node from the live-AWS T3 module.""" + proc = _run_pytest(["--collect-only", "-q", "tests/e2e", "tests/integration"], tmp_path) + assert proc.returncode in (0, 5), proc.stdout + proc.stderr + nodes = _node_ids(proc.stdout) + live_nodes = [n for n in nodes if "test_agentcore_live_e2e" in n] + assert live_nodes == [], f"T3 live nodes leaked into the default run: {live_nodes}" + # The fully integration-marked roundtrip module must be deselected too. + marked_nodes = [n for n in nodes if "test_agentcore_roundtrip" in n] + assert marked_nodes == [], f"integration-marked nodes in default run: {marked_nodes}" + if _e2e_has_test_modules(): + assert proc.returncode == 0, proc.stdout + proc.stderr + assert any(n.startswith("tests/e2e/") for n in nodes), ( + f"no tests/e2e nodes in default collection:\n{proc.stdout}" + ) + + +def test_e2e_marker_is_declared(tmp_path: Path) -> None: + """`pytest --markers` is pytest's own view of declared markers — red the + moment someone deletes the pyproject declaration.""" + proc = _run_pytest(["--markers"], tmp_path) + assert proc.returncode == 0, proc.stdout + proc.stderr + assert "e2e: end-to-end clean-slate smoke tests (hermetic)" in proc.stdout + # The pre-existing marker must survive the addition. + assert "integration:" in proc.stdout + + +def test_e2e_marked_collection_emits_no_unknown_mark_warning(tmp_path: Path) -> None: + proc = _run_pytest(["--collect-only", "-q", "-m", "e2e", "tests/e2e"], tmp_path) + combined = proc.stdout + proc.stderr + assert "PytestUnknownMarkWarning" not in combined, combined + assert "Unknown pytest.mark.e2e" not in combined, combined + assert proc.returncode in (0, 5), combined + if _e2e_has_test_modules(): + # conftest auto-marking guarantees every tests/e2e test carries e2e. + assert proc.returncode == 0, combined + assert len(_node_ids(proc.stdout)) > 0, combined + + +def test_t3_nodes_reachable_under_integration_marker(tmp_path: Path) -> None: + """T3 exists and is collectable when explicitly requested (step 3).""" + if not LIVE_T3.exists(): + pytest.skip("tests/integration/test_agentcore_live_e2e.py not implemented yet (task 12)") + proc = _run_pytest( + [ + "--collect-only", + "-q", + "-m", + "integration", + "--override-ini", + "addopts=", + LIVE_T3.relative_to(REPO_ROOT).as_posix(), + ], + tmp_path, + ) + assert proc.returncode == 0, proc.stdout + proc.stderr + assert len(_node_ids(proc.stdout)) > 0, proc.stdout + + +def test_integration_without_env_gate_is_green_skipped(tmp_path: Path) -> None: + """`-m integration` on a credential-less box must SKIP, never fail and + never exit-5: the gate must stay a pytest.skip (an assert-based refactor + flips this red on every machine without AWS).""" + target = LIVE_T3 if LIVE_T3.exists() else ROUNDTRIP + proc = _run_pytest( + [ + "-q", + "-m", + "integration", + "--override-ini", + "addopts=", + target.relative_to(REPO_ROOT).as_posix(), + ], + tmp_path, + timeout=300, + ) + out = proc.stdout + proc.stderr + # rc 0: not red (gate failed), not exit-5 (vacuously collected nothing). + assert proc.returncode == 0, out + summary = next( + ( + line + for line in reversed(proc.stdout.splitlines()) + if re.search(r"\d+ (skipped|passed|failed|error)", line) + ), + "", + ) + assert re.search(r"\d+ skipped", summary), f"expected skips in summary: {summary!r}\n{out}" + assert "failed" not in summary, summary + assert "error" not in summary, summary diff --git a/tests/hooks/test_observer.py b/tests/hooks/test_observer.py index 646e4ea..406a973 100644 --- a/tests/hooks/test_observer.py +++ b/tests/hooks/test_observer.py @@ -10,6 +10,7 @@ import json import os +import re import subprocess import sys from pathlib import Path @@ -57,12 +58,19 @@ def test_observer_writes_file_for_valid_json(tmp_spool: Path) -> None: result = _run_observer(tmp_spool, stdin=json.dumps(payload)) assert result.returncode == 0, result.stderr + # Observer runs async under pythonw in prod — any stdout print indicates + # a contract misunderstanding. + assert result.stdout == "" files = list(tmp_spool.glob("*.json")) assert len(files) == 1 written = json.loads(files[0].read_text(encoding="utf-8")) assert written["event_type"] == "tool_use" assert written["tool"] == "Edit" assert written["file"] == "auth.py" + # The observer's whole design point is zero-latency, zero-coupling + # spooling: the happy path must never touch (or create) a database + # anywhere under BETTER_MEMORY_HOME. + assert not list(tmp_spool.parent.rglob("*.db")) def test_observer_filename_contains_tool_and_json_ext(tmp_spool: Path) -> None: @@ -81,6 +89,9 @@ def test_observer_filename_contains_tool_and_json_ext(tmp_spool: Path) -> None: assert "Bash" in name # ":" is a reserved char on NTFS; must have been scrubbed from the ts. assert ":" not in name + # Full naming contract: {timestamp}_{tool}_{12-hex-hash}.json. The regex + # tolerates any timestamp encoding while pinning the __<12-hex> tail. + assert re.fullmatch(r".+_Bash_[0-9a-f]{12}\.json", name), name def test_observer_defaults_event_type_when_missing(tmp_spool: Path) -> None: diff --git a/tests/integration/test_agentcore_live_e2e.py b/tests/integration/test_agentcore_live_e2e.py new file mode 100644 index 0000000..44a4c22 --- /dev/null +++ b/tests/integration/test_agentcore_live_e2e.py @@ -0,0 +1,387 @@ +"""T3 live-AWS e2e journey (design scenarios E1-E2). + +Design: docs/superpowers/specs/2026-07-12-e2e-clean-slate-smoke-design.md +section 1E. Gated exactly like the rest of ``tests/integration``: + +* ``@pytest.mark.integration`` (deselected by default via pyproject addopts) +* skip unless ``BETTER_MEMORY_TEST_AGENTCORE=1`` (E2 legs inherit the gate + from the session-scoped ``agentcore_throwaway_memories`` fixture in + ``tests/integration/conftest.py``; E1 gates explicitly because it + provisions its own memories through the real ``agentcore init`` CLI). + +Real AWS credentials come from boto3's default discovery chain in the +outer environment. Region from ``BETTER_MEMORY_TEST_AGENTCORE_REGION`` +(default eu-west-2, via the ``agentcore_region`` conftest fixture). + +E1 — ``e2e-live-init-status-journey``: the only live coverage of the +``init -> agentcore.json -> status`` operator journey (the unit tests are +fully mocked, and the conftest fixture deliberately bypasses the init CLI). +Catches real-AWS drift in the strategy/indexedKeys shape, the ACTIVE poll +loop, and the refuse-to-clobber money-safety contract. + +E2 — ``e2e-live-smoke-retrieve-backend-roundtrip`` (rebuilt per both +judges): (a) the shipped ``agentcore smoke`` 6-step data-plane loop against +real AWS; (b) live MCP ``memory.retrieve`` through the real server — the +one MCP data path actually wired to AgentCoreBackend; (c) direct +``AgentCoreBackend.observe -> list_observations`` read-after-write with +metadata survival. The author-round MCP observe->retrieve_observations +round-trip was REMOVED: in agentcore mode those tools dispatch to local +sqlite (the dispatch gap, design section 4 item 2), so it would greenwash +AWS coverage forever. +""" + +from __future__ import annotations + +import argparse +import asyncio +import json +import os +import subprocess +import sys +import uuid +from datetime import timedelta +from pathlib import Path + +import pytest + +from better_memory.cli import agentcore as agentcore_cli +from better_memory.storage.agentcore_persistence import ( + AgentCoreConfig, + load_agentcore_config, + save_agentcore_config, +) +from tests.e2e._env import isolated_env +from tests.e2e.conftest import mcp_session + +pytestmark = [pytest.mark.integration] + + +def _require_live() -> None: + """Same gate wording/mechanism as tests/integration/conftest.py — MUST + stay a pytest.skip (never assert): ``-m integration`` on a + credential-less box has to end green-skipped (meta-marker-tier-wiring).""" + if os.environ.get("BETTER_MEMORY_TEST_AGENTCORE") != "1": + pytest.skip("Set BETTER_MEMORY_TEST_AGENTCORE=1 to run real-AWS tests.") + + +def _live_env(tmp_home: Path, **pins: str | None) -> dict[str, str]: + """Hermetic-home env with real AWS credentials passed back through. + + Built on the shared ``isolated_env`` choke point (fake HOME/USERPROFILE, + outer CLAUDE_*/BETTER_MEMORY_*/OLLAMA_* stripped) — but live tests need + boto3's default credential chain, which ``isolated_env`` strips along + with everything else AWS_*. Re-pin every outer ``AWS_*`` var verbatim, + and if credentials live only in the real ``~/.aws`` files (no env vars), + point the file-location vars at the real files, because HOME/USERPROFILE + are redirected to ``tmp_home``. + """ + aws_pins: dict[str, str | None] = { + k: v for k, v in os.environ.items() if k.upper().startswith("AWS_") + } + real_aws_dir = Path.home() / ".aws" + upper = {k.upper() for k in aws_pins} + if "AWS_SHARED_CREDENTIALS_FILE" not in upper: + credentials = real_aws_dir / "credentials" + if credentials.exists(): + aws_pins["AWS_SHARED_CREDENTIALS_FILE"] = str(credentials) + if "AWS_CONFIG_FILE" not in upper: + config_file = real_aws_dir / "config" + if config_file.exists(): + aws_pins["AWS_CONFIG_FILE"] = str(config_file) + return isolated_env(tmp_home, **{**aws_pins, **pins}) + + +def _cli_argv(*args: str) -> list[str]: + """Invoke the real CLI dispatcher as a subprocess (module has a + ``__main__`` guard; the ``better-memory`` console script resolves to + the same ``main``).""" + return [sys.executable, "-m", "better_memory.cli.main", *args] + + +# --------------------------------------------------------------------------- +# E1 — e2e-live-init-status-journey +# --------------------------------------------------------------------------- + + +def test_live_init_status_journey( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], + request: pytest.FixtureRequest, + agentcore_region: str, +) -> None: + """``agentcore init`` provisions two ACTIVE memories under throwaway + ``bm_int_*`` names, writes schema-1 agentcore.json, refuses re-init + (money-safety pin), and ``status`` reports both ids ACTIVE. + + init runs IN-PROCESS (``handle(Namespace(...))``) so the monkeypatched + default names apply; status runs as a real subprocess so the whole + json -> get_memory -> ACTIVE aggregation journey is exercised end to + end. Slow (~3-4 min of memory provisioning) but not flaky: unique + suffixes, pre-registered cleanup, and the conftest >1h ``bm_int_`` + stale sweep bound any residue. + """ + _require_live() + import atexit + + import boto3 + from botocore.config import Config as BotoConfig + + # init hardcodes DEFAULT_EPISODIC_NAME / DEFAULT_SEMANTIC_NAME; both are + # imported into cli.agentcore's own namespace and read as module globals + # in _handle_init, so patching cli.agentcore.DEFAULT_* redirects them. + # bm_int_ prefix rides the existing >1h stale-memory sweep; names satisfy + # the AWS regex [a-zA-Z][a-zA-Z0-9_]{0,47}. + suffix = uuid.uuid4().hex[:8] + monkeypatch.setattr(agentcore_cli, "DEFAULT_EPISODIC_NAME", f"bm_int_initepi_{suffix}") + monkeypatch.setattr(agentcore_cli, "DEFAULT_SEMANTIC_NAME", f"bm_int_initsem_{suffix}") + + bm_home = tmp_path / "bm-home" + config_path = bm_home / "agentcore.json" + + control = boto3.client( + "bedrock-agentcore-control", + config=BotoConfig( + region_name=agentcore_region, + retries={"mode": "standard", "max_attempts": 5}, + ), + ) + + # Cleanup registered BEFORE init runs (atexit + finalizer, mirroring the + # conftest fixture pattern) so Ctrl-C / hard kill after a successful + # create still deletes both billable memories. If init itself fails + # mid-flight, its own created_ids orphan cleanup plus the bm_int_ sweep + # cover the gap (agentcore.json won't exist yet in that case). + cleaned: list[bool] = [] + + def _cleanup() -> None: + if cleaned or os.environ.get("BETTER_MEMORY_TEST_AGENTCORE_KEEP") == "1": + return + cleaned.append(True) + if not config_path.exists(): + return + try: + raw = json.loads(config_path.read_text(encoding="utf-8")) + except Exception as exc: # noqa: BLE001 — best-effort teardown + print(f"WARN: could not read {config_path} for cleanup: {exc!r}") + return + for kind in ("episodic", "semantic"): + memory_id = (raw.get(kind) or {}).get("memory_id") + if not memory_id: + continue + try: + control.delete_memory(memoryId=memory_id) + except Exception as exc: # noqa: BLE001 — sweep catches leftovers + print( + f"WARN: failed to delete {memory_id}: {exc!r} " + f"(the bm_int_ stale sweep will catch it)" + ) + + atexit.register(_cleanup) + request.addfinalizer(_cleanup) + + # --- init: provisions both memories, writes agentcore.json ------------- + rc = agentcore_cli.handle( + argparse.Namespace( + subcommand="init", home=str(bm_home), region=agentcore_region, force=False + ) + ) + assert rc == 0 + assert config_path.exists() + + cfg = load_agentcore_config(bm_home) + assert cfg is not None + assert cfg.schema_version == 1 + assert cfg.region == agentcore_region + assert cfg.episodic.memory_id + assert cfg.semantic.memory_id + assert cfg.episodic.memory_id != cfg.semantic.memory_id + # strategyId captured from the ACTIVE memory, not a placeholder — + # downstream semantic writes depend on it. + assert cfg.episodic.strategy_id + assert cfg.semantic.strategy_id + + # --- re-init refusal: money-safety pin --------------------------------- + # A silent overwrite would orphan two billable memories. The refusal + # happens BEFORE any AWS client is built, so the file must be + # byte-identical afterwards. + before_bytes = config_path.read_bytes() + capsys.readouterr() # drain init's progress output + rc2 = agentcore_cli.handle( + argparse.Namespace( + subcommand="init", home=str(bm_home), region=agentcore_region, force=False + ) + ) + captured = capsys.readouterr() + assert rc2 == 1 + assert "already exists" in captured.err + assert "force" in captured.err + assert config_path.read_bytes() == before_bytes + + # --- status subprocess: json -> get_memory -> ACTIVE aggregation ------- + # Region comes from cfg.region inside status (no --region passed); + # credentials come from the AWS_* passthrough in _live_env. + proc_home = tmp_path / "status-home" + proc_home.mkdir() + status = subprocess.run( # noqa: S603 — test harness, fixed argv + _cli_argv("agentcore", "status", "--home", str(bm_home)), + env=_live_env(proc_home), + capture_output=True, + text=True, + timeout=120, + ) + assert status.returncode == 0, status.stdout + status.stderr + assert cfg.episodic.memory_id in status.stdout + assert cfg.semantic.memory_id in status.stdout + assert "episodic:" in status.stdout + assert "semantic:" in status.stdout + assert "ACTIVE" in status.stdout + + +# --------------------------------------------------------------------------- +# E2 — e2e-live-smoke-retrieve-backend-roundtrip +# --------------------------------------------------------------------------- + + +def _write_throwaway_config( + throwaway_memories: tuple, region: str, bm_home: Path +) -> AgentCoreConfig: + """The throwaway MemoryRecords ARE the config — no second init needed.""" + sem_record, epi_record = throwaway_memories + cfg = AgentCoreConfig( + schema_version=1, + region=region, + semantic=sem_record, + episodic=epi_record, + ) + save_agentcore_config(cfg, bm_home) + return cfg + + +def test_live_smoke_cli_passes( + tmp_path: Path, agentcore_throwaway_memories, agentcore_region: str +) -> None: + """E2(a): the shipped ``agentcore smoke`` CLI passes against real AWS. + + Pins the full 6-step data-plane loop (create_event x2 incl. the + role=OTHER closure, list_events >= 2, batch_create with metadata, + list readback, batch_delete) against real AWS wire validation — + tests/cli/test_agentcore_smoke.py is fully mocked, so this is the only + execution of that loop against the real service. + """ + bm_home = tmp_path / "bm-home" + _write_throwaway_config(agentcore_throwaway_memories, agentcore_region, bm_home) + + proc_home = tmp_path / "home" + proc_home.mkdir() + proc = subprocess.run( # noqa: S603 — test harness, fixed argv + _cli_argv("agentcore", "smoke", "--home", str(bm_home)), + env=_live_env(proc_home), + capture_output=True, + text=True, + timeout=180, + ) + assert proc.returncode == 0, proc.stdout + proc.stderr + assert "AgentCore smoke PASSED" in proc.stdout + + +async def test_live_mcp_retrieve_wired_path( + tmp_path: Path, agentcore_throwaway_memories, agentcore_region: str +) -> None: + """E2(b): real MCP server boot with real creds; ``memory.retrieve`` live. + + memory.retrieve is the ONE MCP data tool actually wired to + AgentCoreBackend (dispatch gap, design section 4 item 2), so this is the + only end-to-end MCP-through-AWS path that exists. It fires the + per-polarity ListMemoryRecords fan-out (reflections namespace + + metadataFilters) against the real service — live filter/namespace + validation the T2 fakes structurally cannot see. Fresh throwaway + memories hold no reflections (extraction is minutes-async and the smoke + leg's records live under the ``smoke`` actor's namespaces), so the + buckets are exactly empty. + """ + sem_record, epi_record = agentcore_throwaway_memories + home = tmp_path / "home" + home.mkdir() + # isolated_env pins BETTER_MEMORY_HOME=/.better-memory; the factory + # loads agentcore.json from there. + bm_home = home / ".better-memory" + _write_throwaway_config(agentcore_throwaway_memories, agentcore_region, bm_home) + + env = _live_env( + home, + BETTER_MEMORY_STORAGE_BACKEND="agentcore", + # EXPLICIT region: the runtime factory signs with env + # BETTER_MEMORY_AGENTCORE_REGION (default eu-west-2), NOT + # agentcore.json's region — the region split-brain (design section 4 + # item 4, pinned by e2e-ac-region-split-brain-pin). Without this pin + # a non-default test region silently cross-regions the data plane. + BETTER_MEMORY_AGENTCORE_REGION=agentcore_region, + # FIXME(idvar-gate): config.py:293-301 requires both ID vars in + # agentcore mode but nothing consumes their values (IDs come from + # agentcore.json). Set to the REAL throwaway ids — never dummies in a + # live test, so if the product ever starts consuming them they still + # point at the right memories. Delete when the gate is fixed + # (together with tests/e2e/_agentcore_env.py's dummy vars and + # e2e-ac-neg-prehandshake-config-errors[idvar]). + BETTER_MEMORY_AGENTCORE_SEMANTIC_MEMORY_ID=sem_record.memory_id, + BETTER_MEMORY_AGENTCORE_EPISODIC_MEMORY_ID=epi_record.memory_id, + # No dashes: actor ids derive from the project name. + BETTER_MEMORY_PROJECT="bmintproj", + CLAUDE_SESSION_ID=f"bm-int-{uuid.uuid4().hex[:8]}", + ) + + errlog_path = tmp_path / "server.stderr" # outside the fake home + with errlog_path.open("w", encoding="utf-8") as errlog: + async with mcp_session( + env, errlog=errlog, read_timeout=timedelta(seconds=90) + ) as session: + result = await session.call_tool("memory.retrieve", {}) + content = result.content + assert not getattr(result, "isError", False), ( + f"memory.retrieve errored: {content!r}\n" + f"server stderr:\n{errlog_path.read_text(encoding='utf-8', errors='replace')}" + ) + assert len(content) == 1, f"expected one content block: {content!r}" + assert getattr(content[0], "type", None) == "text" + buckets = json.loads(content[0].text) + + # Exact empty buckets: real AWS accepted the namespace + status + # metadataFilters and returned zero reflections for this actor. + assert buckets == {"do": [], "dont": [], "neutral": []} + + +def test_live_backend_observe_metadata_survives_roundtrip(agentcore_backend) -> None: + """E2(c): direct AgentCoreBackend observe -> list_observations with a + unique marker: read-after-write plus METADATA SURVIVAL through the + stringValue flattening (storage/agentcore.py list_observations mapping) + — the assertion tests/integration/test_agentcore_roundtrip.py lacks + (it only checks event-id presence). + + Event read-after-write is promptly consistent (the shipped smoke CLI + relies on it); nothing here waits on async reflection extraction. The + fixture's unique per-test session id means list_observations (current + session only) sees exactly this event. + """ + marker = f"bm-int-e2e-{uuid.uuid4().hex[:8]}" + event_id = asyncio.run( + agentcore_backend.observe( + content=marker, + outcome="success", + theme="integration", + ) + ) + assert isinstance(event_id, str) and event_id + + events = asyncio.run(agentcore_backend.list_observations(limit=50)) + matches = [e for e in events if e.get("content") == marker] + assert len(matches) == 1, ( + f"expected exactly the marker event back, got {len(matches)} matches " + f"among {len(events)} events" + ) + item = matches[0] + assert item["id"] == event_id + # outcome/theme were sent as metadata stringValues on CreateEvent and + # flattened back out of ListEvents — the AWS round-trip must preserve them. + assert item.get("outcome") == "success" + assert item.get("theme") == "integration" diff --git a/tests/storage/test_agentcore_unit.py b/tests/storage/test_agentcore_unit.py index c4c1562..4cde7c2 100644 --- a/tests/storage/test_agentcore_unit.py +++ b/tests/storage/test_agentcore_unit.py @@ -722,7 +722,12 @@ def test_semantic_observe_calls_batch_create_against_semantic_memory(backend, mo assert rec["memoryStrategyId"] == "userPreference-zXy1234567" assert rec["namespaces"] == ["projects/testproj/semantic/"] assert rec["content"]["text"] == "prefer uv over pip" - assert len(rec["requestIdentifier"]) <= 80 + # Content-hash dedup contract: requestIdentifier is sha256(content)[:80], + # computed here independently so this is a genuine oracle. A scheme change + # (uuid4, content+timestamp hash) silently breaks natural dedup of + # repeated preferences on AWS. + expected_req_id = hashlib.sha256(b"prefer uv over pip").hexdigest()[:80] + assert rec["requestIdentifier"] == expected_req_id # Initial metadata assert rec["metadata"]["status"]["stringValue"] == "active" assert rec["metadata"]["useful_count"]["numberValue"] == 0 From 18188134a75f5c1c09fe0b75d606a53235ea28fe Mon Sep 17 00:00:00 2001 From: gethin Date: Sun, 12 Jul 2026 15:28:32 +0100 Subject: [PATCH 3/4] test: fix review-confirmed e2e harness defects MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Validation-phase adversarial review + verification findings: - env-choke-point AST checker: per-scope blessing with taint-on-reassignment (was module-wide param-name blessing — hand-rolled env dicts could slip through), wrapper call-site inspection, attribute-env fill-site policing, os.environ | {...} merge ban; bypass patterns pinned by a new self-test - canary meta-run: pin host uv cache/python dirs into the hostile outer env before the HOME override (POSIX CI would cold-cache into the canary home and report a false isolation breach) - real_home_canary: data-plane tell-tales (episode/exposure rows and spool/runtime files carrying harness-pinned e2e-session*/e2e-project fingerprints) — closes the blind spot where a leaked spawn writes the real memory.db without touching config subtrees - contextual_inject except-path test: assert state/ absent — the schema-less memory.db alone is produced by the happy path too and could not prove the except branch ran (M3 mutation sentinel integrity) - mutation driver: scratch worktree moved off %TEMP% (Git Bash mounts it as /tmp, whose non-drive MSYS form breaks setup.sh win_path in the control run); strip UV_PROJECT_ENVIRONMENT/VIRTUAL_ENV so the worktree venv tests patched sources - refused-Ollama port: bind-ephemeral-then-close instead of hardcoded :9 - agentcore in-process fixture: scrub HTTP(S)_PROXY/ALL_PROXY + NO_PROXY pin - fake endpoint: accept both .json.gz and .json botocore model packaging - SDK negative test: stop pinning mcp-SDK-internal 'Connection closed' prose Co-Authored-By: Claude Fable 5 --- scripts/e2e_mutation_smoke.py | 24 +- tests/e2e/_fake_agentcore.py | 19 +- tests/e2e/conftest.py | 64 +++++- tests/e2e/test_agentcore_neg.py | 5 +- tests/e2e/test_agentcore_t2.py | 9 + tests/e2e/test_hooks_contracts.py | 16 +- tests/e2e/test_sqlite_negative.py | 20 +- tests/e2e_meta/test_canary_home.py | 9 + tests/e2e_meta/test_env_helper_contract.py | 242 ++++++++++++++++----- 9 files changed, 338 insertions(+), 70 deletions(-) diff --git a/scripts/e2e_mutation_smoke.py b/scripts/e2e_mutation_smoke.py index 978f70e..34f7874 100644 --- a/scripts/e2e_mutation_smoke.py +++ b/scripts/e2e_mutation_smoke.py @@ -177,11 +177,33 @@ def _pytest_env(mutation: Mutation | None) -> dict[str, str]: for key in list(env): if key.upper() in _CONTROL_VARS: del env[key] + # The worktree must resolve its OWN editable venv — an inherited + # UV_PROJECT_ENVIRONMENT / VIRTUAL_ENV would silently point `uv run` + # back at the live repo's install and test unpatched code. + env.pop("UV_PROJECT_ENVIRONMENT", None) + env.pop("VIRTUAL_ENV", None) if mutation is not None: env.update(mutation.extra_env) return env +def _scratch_dir() -> Path: + """A short-lived scratch root for the throwaway worktree. + + On Windows this must NOT live under %TEMP%: Git Bash mounts %TEMP% as + /tmp, so setup.sh sees the worktree at /tmp/... — a non-drive-letter + MSYS path that win_path() cannot rewrite, which breaks the two + test_setup_sh interpreter-path assertions in the control run for + environment reasons unrelated to HEAD. LOCALAPPDATA itself (the parent + of %TEMP%) is not mounted at /tmp and keeps paths short. + """ + if sys.platform == "win32": + base = os.environ.get("LOCALAPPDATA") + if base and Path(base).is_dir(): + return Path(tempfile.mkdtemp(prefix="bm-mutsmoke-", dir=base)) + return Path(tempfile.mkdtemp(prefix="bm-mutsmoke-")) + + def _tail(text: str, lines: int = 30) -> str: return "\n".join(text.splitlines()[-lines:]) @@ -419,7 +441,7 @@ def main(argv: list[str] | None = None) -> int: for line in dirty.stdout.strip().splitlines(): _log(f" {line}") - scratch = Path(tempfile.mkdtemp(prefix="bm-mutsmoke-")) + scratch = _scratch_dir() worktree: Path | None = None failures: list[str] = [] try: diff --git a/tests/e2e/_fake_agentcore.py b/tests/e2e/_fake_agentcore.py index 1791a09..a69c24d 100644 --- a/tests/e2e/_fake_agentcore.py +++ b/tests/e2e/_fake_agentcore.py @@ -45,10 +45,14 @@ import botocore -#: Both planes' models, relative to the botocore package dir. The trailing -#: ``.gz`` is load-bearing: the installed models are gzipped (judge fix — -#: a plain ``service-2.json`` glob finds nothing at botocore 1.43.14). -_MODEL_GLOB = "data/bedrock-agentcore*/*/service-2.json.gz" +#: Both planes' models, relative to the botocore package dir. Models are +#: gzipped at botocore 1.43.14 (judge fix), but the packaging has flipped +#: between .json and .json.gz across botocore history — glob both so a +#: botocore upgrade cannot silently break routing (review fix). +_MODEL_GLOBS = ( + "data/bedrock-agentcore*/*/service-2.json.gz", + "data/bedrock-agentcore*/*/service-2.json", +) _PLACEHOLDER_RE = re.compile(r"\{([^}]+)\}") @@ -72,15 +76,16 @@ def _load_routes() -> list[tuple[str, str, re.Pattern[str]]]: """Build ``(operation_name, http_method, path_regex)`` routes from the installed gzipped botocore models for both agentcore planes.""" data_root = Path(botocore.__file__).resolve().parent - model_paths = sorted(data_root.glob(_MODEL_GLOB)) + model_paths = sorted(p for g in _MODEL_GLOBS for p in data_root.glob(g)) if not model_paths: raise FileNotFoundError( f"no bedrock-agentcore service models found under {data_root} " - f"(glob {_MODEL_GLOB!r}) — is the [agentcore] extra installed?" + f"(globs {_MODEL_GLOBS!r}) — is the [agentcore] extra installed?" ) routes: list[tuple[str, str, re.Pattern[str]]] = [] for model_path in model_paths: - model = json.loads(gzip.decompress(model_path.read_bytes())) + raw = model_path.read_bytes() + model = json.loads(gzip.decompress(raw) if model_path.suffix == ".gz" else raw) for op_name, op in model.get("operations", {}).items(): http = op.get("http", {}) request_uri = http.get("requestUri") diff --git a/tests/e2e/conftest.py b/tests/e2e/conftest.py index fad19d7..0aa0305 100644 --- a/tests/e2e/conftest.py +++ b/tests/e2e/conftest.py @@ -26,13 +26,14 @@ import hashlib import json import os +import sqlite3 import subprocess import sys import tempfile import uuid import warnings from collections.abc import AsyncIterator, Iterator -from contextlib import asynccontextmanager +from contextlib import asynccontextmanager, closing from datetime import timedelta from pathlib import Path from typing import IO, Any @@ -262,6 +263,58 @@ def _canary_sha256(path: Path) -> str: return "" +#: Values only the harness ever pins (tests/e2e/_env.py defaults). A live +#: Claude session writes spool files and DB rows constantly, but never with +#: these — fingerprint-filtered tell-tales are live-session-safe. +_HARNESS_FINGERPRINTS = ("e2e-session", "e2e-project") + + +def _canary_dataplane(home: Path) -> dict[str, Any]: + """Data-plane tell-tales in the REAL ~/.better-memory (review finding + isolation-canary-blind-to-dataplane-writes): a leaked hook/server spawn + that kept the real USERPROFILE writes episode/exposure rows and spool or + session-marker files carrying the harness's pinned session/project values. + Config-subtree comparison alone is blind to that.""" + bm = home / ".better-memory" + out: dict[str, Any] = {} + + db = bm / "memory.db" + for key, sql in ( + ( + "db_episode_sessions", + "SELECT COUNT(*) FROM episode_sessions WHERE session_id LIKE 'e2e-session%'", + ), + ( + "db_exposures", + "SELECT COUNT(*) FROM session_memory_exposure WHERE session_id LIKE 'e2e-session%'", + ), + ): + try: + with closing( + sqlite3.connect(f"file:{db.as_posix()}?mode=ro", uri=True, timeout=1) + ) as conn: + (out[key],) = conn.execute(sql).fetchone() + except (sqlite3.Error, OSError): + out[key] = "" # absent DB / locked / pre-migration schema + + hits: list[str] = [] + for sub in ("spool", "runtime/sessions"): + directory = bm / Path(sub) + if not directory.is_dir(): + continue + for p in sorted(directory.iterdir()): + if p.name.startswith(CANARY_PREFIX) or not p.is_file(): + continue + try: + text = p.read_text(encoding="utf-8", errors="replace") + except OSError: + continue + if any(fp in text for fp in _HARNESS_FINGERPRINTS): + hits.append(f"{sub}/{p.name}") + out["fingerprinted_files"] = hits + return out + + def _real_home_snapshot(home: Path) -> dict[str, Any]: """Tell-tale state that only a leaked e2e subprocess would mutate.""" claude_json = _canary_read_json(home / ".claude.json") @@ -273,6 +326,7 @@ def _real_home_snapshot(home: Path) -> dict[str, Any]: "skills": _canary_entry_names(home / ".claude" / "skills"), "claude_json_sha": _canary_sha256(home / ".claude.json"), "settings_sha": _canary_sha256(home / ".claude" / "settings.json"), + "dataplane": _canary_dataplane(home), } @@ -323,6 +377,13 @@ def real_home_canary() -> Iterator[None]: (demoted) hash warning would notice. That case is owned by the whole-suite meta-run in tests/e2e_meta/test_canary_home.py, which runs against a fresh seeded canary home where any leak is loud. + + Data-plane coverage (review fix): the snapshot also fingerprints the + real ``~/.better-memory`` data plane — episode/exposure rows and + spool / runtime-session files carrying the harness-pinned + ``e2e-session*`` / ``e2e-project`` values — so a leaked hook or server + spawn that writes the user's real memory.db no longer goes unnoticed + just because the config subtrees stayed intact. """ home = _PRISTINE_REAL_HOME worker = os.environ.get("PYTEST_XDIST_WORKER", "main") @@ -362,6 +423,7 @@ def real_home_canary() -> Iterator[None]: ("hook_bm", "~/.claude/settings.json better-memory hook entries"), ("install_backups", "~/.better-memory/install-backups entries"), ("skills", "~/.claude/skills entry names"), + ("dataplane", "~/.better-memory data-plane harness fingerprints"), ): if before[key] != after[key]: breaches.append( diff --git a/tests/e2e/test_agentcore_neg.py b/tests/e2e/test_agentcore_neg.py index f9da9e6..2bcd728 100644 --- a/tests/e2e/test_agentcore_neg.py +++ b/tests/e2e/test_agentcore_neg.py @@ -201,7 +201,10 @@ async def test_sdk_client_sees_mcperror_connection_closed( # already dead, so shutdown returns immediately. assert exc_info.value.error.code == -32000 - assert "Connection closed" in exc_info.value.error.message + # The error MESSAGE ('Connection closed') is mcp-SDK internal prose — + # not our contract. Pin only that a message exists; the -32000 code + # plus the product's own stderr text below carry the real assertions. + assert exc_info.value.error.message assert "agentcore.json not found" in errlog_path.read_text( encoding="utf-8" ) diff --git a/tests/e2e/test_agentcore_t2.py b/tests/e2e/test_agentcore_t2.py index c1d52c8..fffe99f 100644 --- a/tests/e2e/test_agentcore_t2.py +++ b/tests/e2e/test_agentcore_t2.py @@ -260,8 +260,17 @@ def scrubbed_aws_process_env( "AWS_SESSION_TOKEN", "AWS_ENDPOINT_URL", "AWS_IGNORE_CONFIGURED_ENDPOINT_URLS", + # botocore resolves env proxies for loopback targets too — a dev + # shell's corporate proxy would swallow the fake-endpoint traffic. + "HTTP_PROXY", + "HTTPS_PROXY", + "ALL_PROXY", + "http_proxy", + "https_proxy", + "all_proxy", ): monkeypatch.delenv(var, raising=False) + monkeypatch.setenv("NO_PROXY", "127.0.0.1,localhost") monkeypatch.setenv( "AWS_SHARED_CREDENTIALS_FILE", str(tmp_path / "no-such-credentials") ) diff --git a/tests/e2e/test_hooks_contracts.py b/tests/e2e/test_hooks_contracts.py index 3577a5e..b36bad0 100644 --- a/tests/e2e/test_hooks_contracts.py +++ b/tests/e2e/test_hooks_contracts.py @@ -108,10 +108,15 @@ def test_a_config_error_swallowed_envelope_still_printed( raise ValueError *inside* the try — the hook must swallow it, record_hook_error, and still print the envelope with exit 0. - Anti-vacuity: record_hook_error's connect() creates a schema-less - memory.db on a virgin home (its INSERT then silently no-ops — same - stray-DB defect pinned at agentcore level in scenario D6). Asserting - that artifact proves the except path actually executed. + Anti-vacuity: the schema-less memory.db alone is NOT discriminating — + the happy path on a virgin home creates an identical file via its own + connect(). The discriminator is ``state/``: SeenStore.bump_turn() + mkdirs it on the happy path (contextual_inject.py sets it up right + after get_config()), but an invalid mode raises inside get_config() + BEFORE that block — so memory.db present + state/ absent proves the + except path actually executed. (This test is the M3 mutation sentinel; + without the state/ assertion it would silently degrade into a + happy-path duplicate if the mode validation were ever loosened.) """ env = isolated_env( clean_slate_home, BETTER_MEMORY_CONTEXT_INJECT_MODE="aggressive" @@ -131,6 +136,9 @@ def test_a_config_error_swallowed_envelope_still_printed( with closing(sqlite3.connect(db)) as conn: (tables,) = conn.execute("SELECT COUNT(*) FROM sqlite_master").fetchone() assert tables == 0 + # state/ is created by the happy path's SeenStore but never reached + # when get_config() raises — the genuinely discriminating assertion. + assert not (clean_slate_home / ".better-memory" / "state").exists() async def test_b_seeded_memory_injects_and_records_contextual_exposure( self, clean_slate_home: Path, tmp_path: Path diff --git a/tests/e2e/test_sqlite_negative.py b/tests/e2e/test_sqlite_negative.py index 27448b7..c07f8c9 100644 --- a/tests/e2e/test_sqlite_negative.py +++ b/tests/e2e/test_sqlite_negative.py @@ -18,6 +18,7 @@ from __future__ import annotations +import socket import sqlite3 import time from contextlib import closing @@ -26,11 +27,20 @@ from tests.e2e._env import isolated_env from tests.e2e.conftest import mcp_session -#: Loopback discard port: no listener on dev/CI boxes, so connects fail with -#: an instant ECONNREFUSED — no DNS, no route, no 30s connect timeout. If a -#: machine ever runs something on port 9 the failure mode is a clear -#: assertion message, not a hang (design C8 determinism note). -OLLAMA_ABSENT_HOST = "http://127.0.0.1:9" + +def _refused_loopback_port() -> int: + """A loopback port with provably no listener: bind an ephemeral port, + close it, use it immediately. Connects fail with an instant + ECONNREFUSED — no DNS, no route, no 30s connect timeout. (Review fix: + the previous hardcoded port 9 is not guaranteed free — Windows' + optional Simple TCP/IP Services runs a discard listener there, which + would turn this test into a hang/false pass.)""" + with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as sock: + sock.bind(("127.0.0.1", 0)) + return sock.getsockname()[1] + + +OLLAMA_ABSENT_HOST = f"http://127.0.0.1:{_refused_loopback_port()}" def _text_of(result: object) -> str: diff --git a/tests/e2e_meta/test_canary_home.py b/tests/e2e_meta/test_canary_home.py index 5c0dd27..bda4911 100644 --- a/tests/e2e_meta/test_canary_home.py +++ b/tests/e2e_meta/test_canary_home.py @@ -92,6 +92,15 @@ def _hostile_outer_env(canary: Path) -> dict[str, str]: # silently turn the meta-run vacuous — drop it. for key in [k for k in env if k.upper() == "PYTEST_ADDOPTS"]: del env[key] + # Pin uv's cache/python dirs to the HOST's real locations BEFORE the HOME + # override below: on POSIX the inner test_setup_sh would otherwise derive + # them from $HOME (== the canary), and setup.sh's `uv sync` would populate + # a cold cache tree inside the canary — a false HARNESS ISOLATION BREACH + # in the file-set diff (plus a multi-minute network download). + from tests.e2e.test_setup_sh import _host_uv_dirs + + for key, value in _host_uv_dirs().items(): + _set_ci(env, key, value) _set_ci(env, "HOME", str(canary)) _set_ci(env, "USERPROFILE", str(canary)) if sys.platform == "win32": diff --git a/tests/e2e_meta/test_env_helper_contract.py b/tests/e2e_meta/test_env_helper_contract.py index 385d3f0..5e74252 100644 --- a/tests/e2e_meta/test_env_helper_contract.py +++ b/tests/e2e_meta/test_env_helper_contract.py @@ -233,22 +233,40 @@ def _func_source(src: str, node: ast.Call) -> str: return (ast.get_source_segment(src, node.func) or "").strip() -def _blessed_names(src: str, tree: ast.AST) -> set[str]: - """Names whose value provably derives from isolated_env/agentcore_env. +def _walk_scope(node: ast.AST) -> list[ast.AST]: + """Nodes of one lexical scope — does NOT descend into nested function, + lambda, or class scopes (each gets analyzed as its own scope).""" + out: list[ast.AST] = [] + for child in ast.iter_child_nodes(node): + if isinstance( + child, ast.FunctionDef | ast.AsyncFunctionDef | ast.Lambda | ast.ClassDef + ): + continue + out.append(child) + out.extend(_walk_scope(child)) + return out + + +def _scope_blessed(src: str, scope: ast.AST) -> set[str]: + """Names in THIS scope provably derived from isolated_env/agentcore_env. - Seeds: function parameters named ``env``/``*_env`` (their values flow - from call sites, which are themselves checked). Fixed point over simple - assignments whose RHS mentions a helper or an already-blessed name. + Seeds: the scope's own ``env``/``*_env`` parameters (their call sites are + checked separately by the wrapper rule). Fixed point over the scope's + simple assignments. TAINT WINS: a name with any assignment whose RHS does + not derive from a helper or blessed name is never blessed — a reassigned + parameter loses its blessing, and a hand-rolled ``env = {...}`` in a test + body is a violation even though some other function has an ``env`` param + (the module-wide-blessing bypass found in review). """ - blessed: set[str] = set() + seeds: set[str] = set() + if isinstance(scope, ast.FunctionDef | ast.AsyncFunctionDef): + a = scope.args + for arg in [*a.posonlyargs, *a.args, *a.kwonlyargs]: + if _ENV_PARAM.match(arg.arg): + seeds.add(arg.arg) assignments: list[tuple[list[str], str]] = [] - for node in ast.walk(tree): - if isinstance(node, ast.FunctionDef | ast.AsyncFunctionDef): - a = node.args - for arg in [*a.posonlyargs, *a.args, *a.kwonlyargs]: - if _ENV_PARAM.match(arg.arg): - blessed.add(arg.arg) - elif isinstance(node, ast.Assign): + for node in _walk_scope(scope): + if isinstance(node, ast.Assign): names = [t.id for t in node.targets if isinstance(t, ast.Name)] if names: assignments.append((names, ast.get_source_segment(src, node.value) or "")) @@ -257,56 +275,143 @@ def _blessed_names(src: str, tree: ast.AST) -> set[str]: assignments.append( ([node.target.id], ast.get_source_segment(src, node.value) or "") ) - changed = True - while changed: - changed = False + tainted: set[str] = set() + blessed: set[str] = set() + for _ in range(10): # small fixed point; taint and blessing interact + new_blessed = set(seeds) - tainted + changed = True + while changed: + changed = False + for names, rhs in assignments: + ok = bool(_HELPER_NAME.search(rhs)) or any( + re.search(rf"\b{re.escape(b)}\b", rhs) for b in new_blessed + ) + if ok: + for n in names: + if n not in new_blessed and n not in tainted: + new_blessed.add(n) + changed = True + new_tainted = set(tainted) for names, rhs in assignments: - if all(n in blessed for n in names): - continue ok = bool(_HELPER_NAME.search(rhs)) or any( - re.search(rf"\b{re.escape(b)}\b", rhs) for b in blessed + re.search(rf"\b{re.escape(b)}\b", rhs) for b in new_blessed ) - if ok: - for n in names: - if n not in blessed: - blessed.add(n) - changed = True + if not ok: + new_tainted.update(names) + new_blessed -= new_tainted + if new_blessed == blessed and new_tainted == tainted: + break + blessed, tainted = new_blessed, new_tainted return blessed +def _env_param_wrappers(tree: ast.AST) -> dict[str, tuple[int, str]]: + """Module-local functions taking an env-like parameter: name -> + (positional index, param name). Their call sites must pass a + helper-derived env — otherwise the parameter seed would launder + arbitrary dicts through the wrapper (review bypass).""" + wrappers: dict[str, tuple[int, str]] = {} + for node in ast.walk(tree): + if isinstance(node, ast.FunctionDef | ast.AsyncFunctionDef): + params = [*node.args.posonlyargs, *node.args.args] + for idx, arg in enumerate(params): + if _ENV_PARAM.match(arg.arg): + wrappers[node.name] = (idx, arg.arg) + break + else: + for arg in node.args.kwonlyargs: + if _ENV_PARAM.match(arg.arg): + wrappers[node.name] = (-1, arg.arg) + break + return wrappers + + +#: The env choke-point modules themselves hand-build the dict they return — +#: their internal _set/_delete calls are the one legitimate exception to the +#: wrapper rule. They still get the subprocess-spawn rules (they spawn nothing +#: today; if one ever does, it must use its own product). +_CHOKE_POINT_MODULES = {"_env.py", "_agentcore_env.py"} + + def _spawn_env_violations(path: Path) -> list[str]: src = path.read_text(encoding="utf-8") tree = ast.parse(src, filename=str(path)) - blessed = _blessed_names(src, tree) + is_choke_point = path.name in _CHOKE_POINT_MODULES + wrappers = {} if is_choke_point else _env_param_wrappers(tree) violations: list[str] = [] - def env_ok(expr: ast.expr | None, where: str) -> None: - if expr is None: - violations.append(f"{path.name}:{where}: spawn without explicit env=") - return - segment = ast.get_source_segment(src, expr) or "" - if _HELPER_NAME.search(segment): - return - if isinstance(expr, ast.Name) and expr.id in blessed: - return - violations.append( - f"{path.name}:{where}: env not derived from isolated_env/agentcore_env: {segment!r}" - ) - + # Attribute-held envs (self.env / harness.env) are acceptable at use sites + # because their fill sites are checked: ctor calls carry env= (generic + # kwarg rule) and explicit `x.env = ...` assignments are policed here. for node in ast.walk(tree): - if not isinstance(node, ast.Call): - continue - func = _func_source(src, node) - kw_env = next((k.value for k in node.keywords if k.arg == "env"), None) - line = f"line {node.lineno}" - if _SUBPROCESS_FUNC.match(func) or func.endswith("StdioServerParameters"): - env_ok(kw_env, line) - elif func.endswith("run_hook"): - pos = node.args[2] if len(node.args) >= 3 else None - env_ok(kw_env if kw_env is not None else pos, line) - elif func.endswith("mcp_session"): - pos = node.args[0] if node.args else None - env_ok(kw_env if kw_env is not None else pos, line) + if isinstance(node, ast.Assign): + for target in node.targets: + if isinstance(target, ast.Attribute) and _ENV_PARAM.match(target.attr): + rhs = ast.get_source_segment(src, node.value) or "" + if not (_HELPER_NAME.search(rhs) or re.search(r"\benv\b|_env\b", rhs)): + violations.append( + f"{path.name}:line {node.lineno}: attribute env assigned " + f"from non-derived value: {rhs!r}" + ) + + scopes: list[ast.AST] = [tree] + scopes.extend( + n for n in ast.walk(tree) if isinstance(n, ast.FunctionDef | ast.AsyncFunctionDef) + ) + for scope in scopes: + scope_blessed = _scope_blessed(src, scope) + + def env_ok( + expr: ast.expr | None, where: str, blessed: set[str] = scope_blessed + ) -> None: + if expr is None: + violations.append(f"{path.name}:{where}: spawn without explicit env=") + return + segment = ast.get_source_segment(src, expr) or "" + if "environ" in segment: + violations.append( + f"{path.name}:{where}: env mentions os.environ: {segment!r}" + ) + return + if _HELPER_NAME.search(segment): + return + if isinstance(expr, ast.Name) and expr.id in blessed: + return + if isinstance(expr, ast.Attribute) and _ENV_PARAM.match(expr.attr): + return # fill sites policed above / via ctor env= kwarg rule + # Inline derivations of a blessed name, e.g. {**env, "X": "1"}. + if any(re.search(rf"\b{re.escape(b)}\b", segment) for b in blessed): + return + violations.append( + f"{path.name}:{where}: env not derived from " + f"isolated_env/agentcore_env: {segment!r}" + ) + + for node in _walk_scope(scope): + if not isinstance(node, ast.Call): + continue + func = _func_source(src, node) + func_tail = func.rsplit(".", 1)[-1] + kw_env = next((k.value for k in node.keywords if k.arg == "env"), None) + line = f"line {node.lineno}" + if _SUBPROCESS_FUNC.match(func) or func.endswith("StdioServerParameters"): + env_ok(kw_env, line) + elif func.endswith("run_hook"): + pos = node.args[2] if len(node.args) >= 3 else None + env_ok(kw_env if kw_env is not None else pos, line) + elif func.endswith("mcp_session"): + pos = node.args[0] if node.args else None + env_ok(kw_env if kw_env is not None else pos, line) + elif func_tail in wrappers: + idx, pname = wrappers[func_tail] + kw = next((k.value for k in node.keywords if k.arg == pname), None) + pos = node.args[idx] if 0 <= idx < len(node.args) else None + arg = kw if kw is not None else pos + if arg is not None: # missing arg = default; wrapper's own body is checked + env_ok(arg, line) + elif kw_env is not None and not is_choke_point: + # Any other call carrying an env= kwarg (harness ctors etc.). + env_ok(kw_env, line) return violations @@ -328,6 +433,12 @@ def _environ_copy_violations(path: Path) -> list[str]: for key, value in zip(node.keys, node.values, strict=True): if key is None and "environ" in (ast.get_source_segment(src, value) or ""): violations.append(f"{path.name}:{node.lineno}: {{**os.environ, ...}}") + elif isinstance(node, ast.BinOp) and isinstance(node.op, ast.BitOr): + # PEP 584 merge: os.environ | {...} is a full-environment copy too. + for side in (node.left, node.right): + if "environ" in (ast.get_source_segment(src, side) or ""): + violations.append(f"{path.name}:{node.lineno}: os.environ | {{...}}") + break return violations @@ -370,6 +481,35 @@ def test_checker_flags_hand_rolled_env_dict(self, tmp_path: Path) -> None: assert any("**os.environ" in v for v in copies) assert any("environ.copy" in v for v in copies) + def test_checker_flags_wrapper_laundering_bypass(self, tmp_path: Path) -> None: + """Review finding isolation-checker-env-param-blessing-bypass: a module + defining a spawn wrapper with an ``env`` param must NOT bless the bare + name 'env' module-wide, and wrapper call sites must be inspected.""" + bad = tmp_path / "test_bypass.py" + bad.write_text( + "import subprocess, sys, os\n" + "def _spawn(env, argv):\n" + " subprocess.run(argv, env=env)\n" + "def test_a(tmp_path):\n" + " env = {'HOME': str(tmp_path)}\n" + " subprocess.run([sys.executable, '-V'], env=env)\n" + "def test_b(tmp_path):\n" + " env = os.environ | {'HOME': str(tmp_path)}\n" + " subprocess.run([sys.executable, '-V'], env=env)\n" + "def test_c(tmp_path):\n" + " _spawn({'HOME': str(tmp_path)}, [sys.executable, '-V'])\n" + "def test_d(tmp_path):\n" + " _spawn(dict(os.environ), [sys.executable, '-V'])\n", + encoding="utf-8", + ) + spawn = _spawn_env_violations(bad) + copies = _environ_copy_violations(bad) + # test_a: hand-rolled name; test_b: tainted name; test_c: inline dict + # at wrapper call; test_d: environ mention at wrapper call. + assert len(spawn) == 4, spawn + assert any("os.environ | {...}" in v for v in copies), copies + assert any("dict(os.environ)" in v for v in copies), copies + def test_checker_accepts_helper_derived_env(self, tmp_path: Path) -> None: good = tmp_path / "test_good.py" good.write_text( From 5048ca3a51a14f56b2bd39248ad58c9a5b5679a9 Mon Sep 17 00:00:00 2001 From: gethin Date: Sun, 12 Jul 2026 15:59:01 +0100 Subject: [PATCH 4/4] test: fix Pyright CI failures in e2e suite - log_message overrides: name the second parameter 'format' to match BaseHTTPRequestHandler's signature (incompatible-override error) - MCP content blocks: CallToolResult.content is a union; add text_of() narrowing helper in tests/e2e/conftest.py and use it at the five direct .text access sites Co-Authored-By: Claude Fable 5 --- tests/e2e/_fake_agentcore.py | 2 +- tests/e2e/conftest.py | 12 ++++++++++++ tests/e2e/test_hooks_contracts.py | 4 ++-- tests/e2e/test_tripwires.py | 10 +++++----- tests/integration/test_agentcore_live_e2e.py | 4 ++-- 5 files changed, 22 insertions(+), 10 deletions(-) diff --git a/tests/e2e/_fake_agentcore.py b/tests/e2e/_fake_agentcore.py index a69c24d..831f3ec 100644 --- a/tests/e2e/_fake_agentcore.py +++ b/tests/e2e/_fake_agentcore.py @@ -183,7 +183,7 @@ def _handle(self) -> None: do_DELETE = _handle do_PATCH = _handle - def log_message(self, *args: Any) -> None: # silence stderr + def log_message(self, format: str, *args: Any) -> None: # noqa: A002 — base class parameter name; silence stderr pass self._server = ThreadingHTTPServer(("127.0.0.1", 0), _Handler) diff --git a/tests/e2e/conftest.py b/tests/e2e/conftest.py index 0aa0305..fcaff5a 100644 --- a/tests/e2e/conftest.py +++ b/tests/e2e/conftest.py @@ -112,6 +112,18 @@ def run_hook( return proc.returncode, proc.stdout, proc.stderr +def text_of(block: object) -> str: + """Text of an MCP content block, narrowing the SDK's content union. + + ``CallToolResult.content`` items are a union (TextContent | ImageContent + | ...); direct ``.text`` access fails type checking. Asserting on the + attribute both narrows for Pyright and fails loudly on a non-text block. + """ + text = getattr(block, "text", None) + assert isinstance(text, str), f"expected a text content block, got: {block!r}" + return text + + @asynccontextmanager async def mcp_session( env: dict[str, str], diff --git a/tests/e2e/test_hooks_contracts.py b/tests/e2e/test_hooks_contracts.py index b36bad0..4fdbf2a 100644 --- a/tests/e2e/test_hooks_contracts.py +++ b/tests/e2e/test_hooks_contracts.py @@ -33,7 +33,7 @@ from typing import Any from tests.e2e._env import isolated_env -from tests.e2e.conftest import mcp_session, run_hook +from tests.e2e.conftest import mcp_session, run_hook, text_of INJECT_HOOK = "better_memory.hooks.contextual_inject" @@ -163,7 +163,7 @@ async def test_b_seeded_memory_injects_and_records_contextual_exposure( "memory.semantic_observe", {"content": content} ) assert not result.isError, result.content - memory_id = json.loads(result.content[0].text)["id"] + memory_id = json.loads(text_of(result.content[0]))["id"] assert memory_id rc, out, err = run_hook( diff --git a/tests/e2e/test_tripwires.py b/tests/e2e/test_tripwires.py index aad91d9..29e3c6c 100644 --- a/tests/e2e/test_tripwires.py +++ b/tests/e2e/test_tripwires.py @@ -22,7 +22,7 @@ from pathlib import Path from tests.e2e._env import isolated_env -from tests.e2e.conftest import mcp_session, run_hook +from tests.e2e.conftest import mcp_session, run_hook, text_of class _RecordingHandler(BaseHTTPRequestHandler): @@ -45,7 +45,7 @@ def _handle(self) -> None: do_HEAD = _handle do_PUT = _handle - def log_message(self, *args: object) -> None: # silence stderr chatter + def log_message(self, format: str, *args: object) -> None: # noqa: A002 — base class parameter name; silence stderr chatter pass @@ -107,7 +107,7 @@ async def test_ollama_zero_traffic_across_journey_and_sync_hooks( {"content": "tripwire observation", "outcome": "success"}, ) assert not r_obs.isError, r_obs.content - obs_id = json.loads(r_obs.content[0].text)["id"] + obs_id = json.loads(text_of(r_obs.content[0]))["id"] assert obs_id # Trigram drill-down (embedder is None in sqlite embeddings @@ -117,7 +117,7 @@ async def test_ollama_zero_traffic_across_journey_and_sync_hooks( "memory.retrieve_observations", {"query": "tripwire"} ) assert not r_drill.isError, r_drill.content - drilled = json.loads(r_drill.content[0].text) + drilled = json.loads(text_of(r_drill.content[0])) matched = [o for o in drilled if o["id"] == obs_id] assert matched, f"observation {obs_id} not surfaced: {drilled!r}" assert "tripwire observation" in matched[0]["content"] @@ -126,7 +126,7 @@ async def test_ollama_zero_traffic_across_journey_and_sync_hooks( # args — reflections filters only, so call it bare). r_buckets = await session.call_tool("memory.retrieve", {}) assert not r_buckets.isError, r_buckets.content - buckets = json.loads(r_buckets.content[0].text) + buckets = json.loads(text_of(r_buckets.content[0])) assert {"do", "dont", "neutral"} <= set(buckets) # --- both synchronous hooks under the same recorder env ----------- diff --git a/tests/integration/test_agentcore_live_e2e.py b/tests/integration/test_agentcore_live_e2e.py index 44a4c22..6a9b82b 100644 --- a/tests/integration/test_agentcore_live_e2e.py +++ b/tests/integration/test_agentcore_live_e2e.py @@ -51,7 +51,7 @@ save_agentcore_config, ) from tests.e2e._env import isolated_env -from tests.e2e.conftest import mcp_session +from tests.e2e.conftest import mcp_session, text_of pytestmark = [pytest.mark.integration] @@ -344,7 +344,7 @@ async def test_live_mcp_retrieve_wired_path( ) assert len(content) == 1, f"expected one content block: {content!r}" assert getattr(content[0], "type", None) == "text" - buckets = json.loads(content[0].text) + buckets = json.loads(text_of(content[0])) # Exact empty buckets: real AWS accepted the namespace + status # metadataFilters and returned zero reflections for this actor.