From f9f66316e582a83ce0de290fde952902e83952a3 Mon Sep 17 00:00:00 2001 From: tbearly Date: Fri, 29 May 2026 14:43:38 -0700 Subject: [PATCH 1/6] Add Card D execution plan Co-Authored-By: Claude Opus 4.8 (1M context) --- execution-plan-card-d.md | 142 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 142 insertions(+) create mode 100644 execution-plan-card-d.md diff --git a/execution-plan-card-d.md b/execution-plan-card-d.md new file mode 100644 index 0000000..df98f89 --- /dev/null +++ b/execution-plan-card-d.md @@ -0,0 +1,142 @@ +# Execution Plan — Card D (Sales Engineer for Product X) + All Stretch Goals + +## Context + +This repo (`institutional-memory-main`) is a **Claude Managed Agents + Memory tool** demo: an agent runs two sessions on the same domain, memory persists between them, session-2 docs contradict session-1, and the agent should reconcile and answer better. The scaffolding (`create_agent.py`, `run_session_1.py`, `run_session_2.py`, `inspect_memory.py`, `stretch_memory_curator.py`) is currently shaped for **Card A — New-Hire Onboarding**, and `synthetic-data/round1`+`round2` hold onboarding docs. + +We are switching to **Card D — Sales Engineer for Product X** and **replacing** the onboarding scenario in place. Product X = **Helios**, a fictional real-time data-streaming platform; the prospect being pitched is **Northwind Bank** (fintech). Goal: a working Card-D demo plus **all nine stretch goals (S1–S9)**. + +**Card D specifics (from `scenario-cards.md`):** +- Persona: agent supporting a sales engineer pitching Helios; it learns the customer's environment, their objections, and our latest answers. +- Round 1 docs: customer stack overview, list of objections from previous calls, current pitch deck. +- Round 2 docs: a new objection from the latest call, a competitive update from product management. +- Test question (both sessions): **"Tomorrow's call is the final pitch. What's our strategy?"** +- "Better in session 2" = anticipates the new objection, references the latest competitive update, and adjusts the pitch *sequence* rather than repeating the original plan. + +**API facts grounded during planning:** +- Managed Agents = `client.beta.agents.create` / `environments.create` / `sessions.create` / `sessions.events.stream`/`send`; memory via `client.beta.memory_stores.create` + `memories.list(store_id, path_prefix=...)` + `memories.retrieve(id, memory_store_id=...)`. Memory is **path-organized** inside a store. +- Compaction is built into the Managed Agents harness; the Messages-API form is `context_management={"edits":[{"type":"compact_20260112"}]}` with beta header `compact-2026-01-12`. +- There is **no documented standalone "Routines" API** (the routines doc 404s). S6 therefore uses a scheduled-wrapper approach (OS scheduler), with a note to prefer a native routines API if one exists in the SDK at execution time. + +--- + +## Part 1 — Core Build (Card D) + +### 1a. Replace synthetic data +**Overwrite** `synthetic-data/round1/` (3 docs) and `synthetic-data/round2/` (2 docs). Match the existing markdown style (dated headers, "effective" dates, tables) so contradictions are detectable by date. + +`synthetic-data/round1/`: +- `helios-stack-overview.md` — Northwind Bank's current environment: Kafka self-managed on-prem, AWS for analytics, strict data-residency (EU + US), SOC2/PCI, peak ~2M events/sec, Flink for stream processing, a 6-person platform team. +- `objections-log.md` — objections raised in prior calls, each with *our current best answer*: (1) latency SLA vs self-managed Kafka, (2) data residency / on-prem requirement, (3) total cost vs existing Kafka, (4) migration risk, (5) vendor lock-in. +- `pitch-deck.md` — current pitch sequence: problem → Helios architecture → managed-vs-self-managed TCO → security/residency → migration path → pricing → next steps. + +`synthetic-data/round2/`: +- `new-objection-2026-05-22.md` — newest call surfaced a **new** objection: Northwind's new CISO mandates *active-active multi-region failover with <5s RPO*, which the current pitch doesn't address. +- `competitive-update-2026-05-25.md` — product management update: competitor **StreamCorp** just cut prices ~20% and announced a managed connector Helios lacks; **but** Helios shipped native active-active multi-region (directly answers the new objection) and a residency-pinning feature. + +### 1b. Repoint the agent + sessions +- **`create_agent.py`** — rewrite `SYSTEM_PROMPT` to the Helios Sales-Engineer persona: "You support a sales engineer pitching Helios to a specific prospect. Track the prospect's environment, every objection + our latest best answer, competitive intel, and the current pitch strategy. Get sharper each call." Keep the existing memory-protocol section. Update `name=` and `metadata` to the SE scenario. (Also receives S1 + S3 additions below.) +- **`run_session_1.py`** — set `TEST_QUESTION` to the Card-D question; reword `user_message` from "onboarding and policy documents" to "the prospect's stack overview, objection log, and current pitch deck." `DOCS_DIR` stays `round1`. +- **`run_session_2.py`** — same `TEST_QUESTION`; reword `user_message` to "a new objection from the latest call and a competitive update from product." `DOCS_DIR` stays `round2`. +- `inspect_memory.py` needs no change (store-agnostic). + +### 1c. Run + compare +`python create_agent.py` → `run_session_1.py` → `inspect_memory.py` → `run_session_2.py`. Diff `outputs/session1.txt` vs `outputs/session2.txt`: session 2 should anticipate the active-active objection, cite the StreamCorp price cut + Helios's new capability, and resequence the pitch. + +--- + +## Part 2 — Stretch Goals (all nine) + +### S1 — Explicit memory policy *(edit `create_agent.py`)* +Add an **ALWAYS remember** list (prospect environment facts; each objection + our best/latest answer; competitive intel with dates; pitch-strategy decisions; key prospect contacts & roles) and a **NEVER remember** list (verbatim deck text — the deck is the source of truth; one-off scheduling/logistics; unconfirmed/speculative pricing; personal data beyond business contact). Recreate the agent and re-run both sessions; memory store should be tighter. + +### S2 — Memory Curator sub-agent *(reuse `stretch_memory_curator.py`)* +Already implemented and largely persona-agnostic. Light tweak to `CURATOR_SYSTEM_PROMPT`: add "merge superseded objection answers; keep only the latest competitive intel per competitor." Run `python stretch_memory_curator.py` after session 2, then re-run session 2 to show a sharper answer. + +### S3 — Adversarial round *(new `synthetic-data/round3/` + `run_session_3.py`)* +Create `round3/` docs that contradict round1 **with no plausible reason / no effective date** (e.g., `bad-stack-overview.md` claiming Northwind is 100% on-prem with no AWS and only 50k events/sec, and a doc claiming the latency objection "was never raised"). Add `run_session_3.py` (copy of `run_session_2.py`, pointed at `round3`, prompt = "reconcile these against memory"). **Expected correct behaviour: flag + ask, not silent overwrite.** If it silently overwrites, harden `create_agent.py`: add a rule — "If a new document contradicts memory but carries no effective date or stated reason, FLAG the conflict and ASK which to trust; do not overwrite." Re-create + re-test. + +### S4 — "What have you learned?" session *(new `run_session_learned.py`)* +Copy session scaffolding but attach **no new docs**; single user message: *"Summarise everything you've learned about this domain across our previous sessions."* Save to `outputs/session_learned.txt`. This is the memory store "talking back." + +### S5 — Per-tenant memory via `customer_id` *(modify create + session scripts)* +Run two prospects — **Northwind Bank** and a second prospect **Globex Retail** — with isolated memory. Approach (recommended for guaranteed no-leak): **one memory store per `customer_id`**, recorded in `.memory_store_` files, with `customer_id` also set in `sessions.create(... metadata=...)` and in the store name/description. Add a `--customer` arg to `create_agent.py` (store creation) and the run scripts; load a small per-customer doc set. Run two sessions each for Northwind and Globex, then verify (via `inspect_memory.py` per store + a cross-question) that Northwind facts never appear in Globex answers. *(Lighter alternative if a single store is required: scope by path prefix `/customers//…` and pass `path_prefix` to `memories.list`.)* + +### S6 — Memory + scheduled ingestion ("Routines") *(new `ingest_new_docs.py` + `synthetic-data/inbox/`)* +No native Routines API is documented. Build `ingest_new_docs.py`: scans `synthetic-data/inbox/` for unseen `.md` files, runs a short session that ingests each into memory, then moves processed files to `synthetic-data/processed/`. Schedule it externally — document the **Windows Task Scheduler** command (and an equivalent cron line) to fire daily. Memory then accumulates passively. Note in code/comments: if `client.beta.routines.*` exists in the installed SDK at run time, prefer wrapping the session in that instead. + +### S7 — Long-context + compaction *(new `run_session_longcontext.py`)* +Drive **20+ turns in a single session** (a simulated long technical Q&A with Northwind: latency, residency, failover, connectors, pricing, migration, security review…), sending each turn via `sessions.events.send` and streaming the reply. Rely on the harness's built-in compaction for in-session sharpness; **final turn instructs the agent to write a compacted summary of the whole call into `/mnt/memory/`**. Comment the distinction: in-session memory (context window + compaction) vs across-session memory (the memory store). If demonstrating the Messages-API form too, reference `context_management={"edits":[{"type":"compact_20260112"}]}` + beta `compact-2026-01-12`. + +### S8 — Memory diff view *(new `memory_snapshot.py` + `memory_diff.py`)* +`memory_snapshot.py