test: clean-slate e2e smoke suite (sqlite + agentcore, agent-team built)#78
Conversation
Design for three-tier e2e smoke suite: T1 hermetic sqlite, T2 hermetic agentcore via AWS_ENDPOINT_URL fake, T3 gated live AWS. Produced by multi-agent workflow: 6 empirical verifiers, 5 scenario authors, 2 adversarial judges per set (60 keep / 40 fix / 7 kill), final synthesis. Documents 12 product defects pinned as current behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…T3 live-gated) Implements the phase-2 design (docs/superpowers/specs/2026-07-12-e2e-clean-slate-smoke-design.md): - tests/e2e/_env.py: isolated_env allowlist choke point (HOME+USERPROFILE+ BETTER_MEMORY_HOME redirect, case-insensitive strip of CLAUDE_*/ BETTER_MEMORY_*/AWS_*/OLLAMA_*) - tests/e2e/conftest.py: clean_slate_home, run_hook, mcp_session (explicit USERPROFILE/HOME for the mcp SDK Windows force-inherit), real_home_canary autouse tripwire, e2e marker auto-application - installer, setup.sh, sqlite journey, hook contracts, sqlite negatives, ollama zero-traffic tripwire scenarios - agentcore hermetic tier: _fake_agentcore.py (botocore-model-routed local endpoint via AWS_ENDPOINT_URL), _agentcore_env.py (single FIXME(idvar-gate) dummy-var location), T2 positives, negatives, AWS lockdown tripwire - tests/e2e_meta: env-helper contract (AST spawn-site provenance), marker/tier wiring, canary-home meta-run, env-bleed poisoned-shell - tests/integration/test_agentcore_live_e2e.py: T3 live journey (gated behind BETTER_MEMORY_TEST_AGENTCORE=1 + integration marker) - scripts/e2e_mutation_smoke.py + M1-M4: suite-power validation gate - 12 known product defects pinned as current behavior with FIXME docstrings Full suite: 1312 passed, 22 skipped, 21 deselected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Validation-phase adversarial review + verification findings:
- env-choke-point AST checker: per-scope blessing with taint-on-reassignment
(was module-wide param-name blessing — hand-rolled env dicts could slip
through), wrapper call-site inspection, attribute-env fill-site policing,
os.environ | {...} merge ban; bypass patterns pinned by a new self-test
- canary meta-run: pin host uv cache/python dirs into the hostile outer env
before the HOME override (POSIX CI would cold-cache into the canary home
and report a false isolation breach)
- real_home_canary: data-plane tell-tales (episode/exposure rows and
spool/runtime files carrying harness-pinned e2e-session*/e2e-project
fingerprints) — closes the blind spot where a leaked spawn writes the
real memory.db without touching config subtrees
- contextual_inject except-path test: assert state/ absent — the schema-less
memory.db alone is produced by the happy path too and could not prove the
except branch ran (M3 mutation sentinel integrity)
- mutation driver: scratch worktree moved off %TEMP% (Git Bash mounts it as
/tmp, whose non-drive MSYS form breaks setup.sh win_path in the control
run); strip UV_PROJECT_ENVIRONMENT/VIRTUAL_ENV so the worktree venv tests
patched sources
- refused-Ollama port: bind-ephemeral-then-close instead of hardcoded :9
- agentcore in-process fixture: scrub HTTP(S)_PROXY/ALL_PROXY + NO_PROXY pin
- fake endpoint: accept both .json.gz and .json botocore model packaging
- SDK negative test: stop pinning mcp-SDK-internal 'Connection closed' prose
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
🟢 Claude BugBot Analysis
Reviewed all 30 changed files (a large new hermetic e2e/e2e_meta test suite plus small additions to existing tests and pyproject.toml). No product code is modified except via the three mutation-smoke patch fixtures, which were verified to apply cleanly and accurately against the current source of better_memory/mcp/server.py, better_memory/hooks/session_bootstrap.py, and better_memory/hooks/contextual_inject.py. The env-isolation helpers, fake AgentCore HTTP server, and mutation-smoke driver were checked for logic errors (case-insensitive env handling, route matching, pytest-output parsing) and found consistent. No genuine bugs found in the added/modified lines.
No bugs were detected in this PR.
- log_message overrides: name the second parameter 'format' to match BaseHTTPRequestHandler's signature (incompatible-override error) - MCP content blocks: CallToolResult.content is a union; add text_of() narrowing helper in tests/e2e/conftest.py and use it at the five direct .text access sites Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟢 Claude BugBot Analysis
This PR adds a large hermetic e2e/e2e_meta test suite (fixtures, fake AWS AgentCore server, mutation-testing gate, canary/tripwire meta-tests) plus minor additions to existing unit tests. After manually reviewing the core infrastructure (env allowlist builder, fake AgentCore HTTP server, conftest fixtures, mutation-smoke driver, mutation patches) and dispatching parallel deep-verification passes over every remaining test file — several of which executed the real test suite and performed targeted mutation testing against the underlying source to rule out vacuous assertions — no genuine bugs, logic errors, security issues, or resource leaks were found in the added/modified lines.
No bugs were detected in this PR.
1 participant
Summary
Three-tier e2e smoke suite proving a brand-new user (no
~/.better-memory, no MCP registration, no hooks, no DB) can set up and use better-memory — without any test touching real user data, real AWS, or the network.AWS_ENDPOINT_URLseam, routed from botocore service models), wire fidelity, negatives, AWS lockdown tripwireBETTER_MEMORY_TEST_AGENTCORE=1 pytest -m integration): agentcore init/status/smoke journey with throwaway memoriesHarness safety:
isolated_envallowlist choke point (AST-enforced — no spawn can bypass it), autouse real-home canary with config + data-plane fingerprint telltales, whole-suite canary meta-run, env-bleed test, mutation-smoke gate (scripts/e2e_mutation_smoke.py).Design doc:
docs/superpowers/specs/2026-07-12-e2e-clean-slate-smoke-design.md(produced by a multi-agent verify → author → adversarial-judge → synthesize pipeline; 60 keep / 40 fix / 7 kill).Validation evidence
1818813Product defects discovered (pinned as current behavior, FIXME docstrings)
memory.observe/semantic_observe/record_usewrite local sqlite, never AWS; onlymemory.retrieve+ hooks reach the backendconfig.py:293-301) — documented agentcore setup dies pre-handshake; the required env vars are consumed by nothingwin_pathbreaks for repos under non-drive MSYS mounts (/tmp, /home)better-memory[agentcore]hintFull list of 12 in design doc §4. Each pinning test names its deletion/inversion condition, so fixing a defect flips the pin loudly.
🤖 Generated with Claude Code