tests: add Session/Memory/Summary multi-backend replay consistency testing framework

docs/replay_harness/DESIGN.md

@@ -0,0 +1,133 @@
+# Session / Memory / Summary Replay Consistency Harness — Design
+
+## Objective
+
+Validate that InMemory, SQL, and Redis backends for Session, Memory, and Summary
+services produce equivalent results when driven by identical operation sequences.
+This harness replays standardized input trajectories through multiple backends
+and generates a structured diff report.
+
+## Architecture
+
+```
+replay_cases/*.json   →  ReplayEngine  →  BackendResult
+                             │
+                    ┌─────────┴─────────┐
+              Backend A            Backend B
+              (InMemory)           (SQL / Redis)
+                    │                    │
+              BackendResult        BackendResult
+                    │                    │
+                    └────────┬───────────┘
+                             ▼
+                      _normalizer.py
+                      (strip timestamps, IDs, sort keys)
+                             │
+                    NormalizedResult A + B
+                             │
+                             ▼
+                      _comparator.py
+                      (pairwise diff events, state, memory, summary)
+                             │
+                             ▼
+                      DiffReport (JSON)
+```
+
+Each replay case is a JSON file describing a sequence of operations:
+`create_session`, `append_event`, `update_state`, `inject_summary`,
+`store_memory`, `search_memory`, `read_back`.
+
+The ReplayEngine executes the same sequence against two backends in parallel,
+collecting raw results. These are then normalized and compared.
+
+## Normalization Strategy
+
+Fields that differ across backends by design are normalized before comparison:
+
+| Field                  | Strategy                                                 |
+|------------------------|----------------------------------------------------------|
+| `event.id`             | Stripped (auto-generated UUID per backend)               |
+| `event.timestamp`      | Replaced with sequential index (0, 1, 2, ...)            |
+| `session.last_update_time` | Replaced with sentinel `0.0`                          |
+| `summary_timestamp`    | Replaced with sentinel `0.0`                             |
+| `memory_entry.timestamp` | Stripped                                               |
+| dict key ordering      | Re-serialized with `sort_keys=True`                      |
+| `invocation_id`        | Stripped (invocation-scoped, not backend-scoped)         |
+| `branch`, `request_id` | Stripped (runtime metadata, not persisted uniformly)     |
+
+## Summary Comparison Strategy
+
+The harness distinguishes two levels of summary correctness:
+
+1. **Content semantic consistency** — the `summary_text` is compared after
+   whitespace normalization. Minor formatting differences that do not change
+   meaning are treated as allowed diffs on a per-backend-pair basis.
+
+2. **Metadata integrity** — `session_id`, `original_event_count`, and
+   `compressed_event_count` must match exactly across backends. Any deviation
+   in these fields is an unconditional failure. Three classes of summary bug
+   are explicitly detected:
+   - **Summary loss** — summary present in backend A, absent in backend B.
+   - **Summary overwrite** — summary exists but with wrong `session_id`.
+   - **Wrong session affiliation** — summary is stored under the wrong session
+     key in the underlying cache or storage.
+
+Summary injection into the `SummarizerSessionManager._summarizer_cache`
+bypasses the LLM, since the harness tests storage consistency, not
+summarization model quality.
+
+## Allowed Differences
+
+Not all field-level differences indicate a bug. Known, documented divergences
+are captured in `AllowedDiff` rules:
+
+```
+allowed_diffs = {
+    "inmem_vs_sql": [
+        {"field": "events[*].function_calls[*].args", "reason": "SQL serializes
+         JSON args differently from InMemory dict round-trip"},
+    ],
+    "inmem_vs_redis": [
+        {"field": "events[*].timestamp_precision",
+         "reason": "Redis stores timestamps as float strings with limited precision"},
+    ],
+}
+```
+
+Each rule includes the backend pair, the field path, and a justification.
+Diffs matching an allowed rule are suppressed from the failure count but
+still appear in the report tagged `allowed: true`.
+
+## Backend Integration
+
+| Backend  | Availability     | Activation                                 |
+|----------|-----------------|---------------------------------------------|
+| InMemory | Always           | Direct instantiation                        |
+| SQL      | Always (sqlite)  | `sqlite:///:memory:` — no external deps     |
+| Redis    | Opt-in           | `TRPC_REDIS_URL` env var; skipped otherwise |
+
+The lightweight mode (InMemory + SQL) runs in CI without external services.
+Redis integration mode requires the environment variable to be set; when
+absent, Redis test pairs are skipped with a descriptive message.
+
+## Diff Report
+
+The report (`session_memory_summary_diff_report.json`) contains:
+
+- **run metadata**: run ID, timestamp, backends tested.
+- **per-case results**: status (pass/fail/error), list of `DiffEntry` objects.
+- **summary**: total/pass/fail counts and false-positive rate.
+
+Each `DiffEntry` pinpoints: backend pair, session ID, event index (or summary
+ID), category (events/state/memory/summary), full dotted field path, and the
+two conflicting values.
+
+## Testing the Harness Itself
+
+- `test_replay_normalizer.py` — unit tests for every normalization function.
+- `test_replay_comparator.py` — unit tests for diff logic, including targeted
+  tests for summary loss, overwrite, and mis-affiliation detection.
+- `test_replay_report.py` — unit tests for report generation and serialization.
+- `test_replay_consistency.py` — E2E tests running all 10 replay cases through
+  backend pairs, asserting normal cases produce ≤5% false positives and anomaly
+  cases are 100% detected.

docs/replay_harness/DESIGN.zh_CN.md

@@ -0,0 +1,111 @@
+# Session / Memory / Summary 回放一致性测试框架 — 设计说明
+
+## 目标
+
+验证 InMemory、SQL、Redis 三种后端的 Session、Memory 和 Summary 服务在相同操作序列下是否产出一致的结果。该框架使用标准化输入轨迹驱动多个后端，并生成结构化差异报告。
+
+## 架构
+
+```
+replay_cases/*.json   →  ReplayEngine  →  BackendResult
+                             │
+                    ┌─────────┴─────────┐
+              后端 A              后端 B
+              (InMemory)          (SQL / Redis)
+                    │                    │
+              BackendResult        BackendResult
+                    │                    │
+                    └────────┬───────────┘
+                             ▼
+                      _normalizer.py
+                      （去除时间戳、ID，排序键）
+                             │
+                    NormalizedResult A + B
+                             │
+                             ▼
+                      _comparator.py
+                      （逐对比较事件、状态、记忆、摘要）
+                             │
+                             ▼
+                      DiffReport（JSON 差异报告）
+```
+
+每个回放用例是一个 JSON 文件，描述一系列操作：
+`create_session`、`append_event`、`update_state`、`inject_summary`、
+`store_memory`、`search_memory`、`read_back`。
+
+ReplayEngine 将相同的操作序列在两个后端上并行执行，收集原始结果后进行归一化和比较。
+
+## 归一化策略
+
+对后端间必然不同的字段，在比较前进行归一化：
+
+| 字段                      | 策略                                          |
+|---------------------------|-----------------------------------------------|
+| `event.id`                | 去除（各后端自动生成 UUID）                     |
+| `event.timestamp`         | 替换为顺序索引（0, 1, 2, ...）                  |
+| `session.last_update_time` | 替换为哨兵值 `0.0`                             |
+| `summary_timestamp`       | 替换为哨兵值 `0.0`                             |
+| `memory_entry.timestamp`  | 去除                                          |
+| dict 键顺序               | 使用 `sort_keys=True` 重新序列化                |
+| `invocation_id`           | 去除（调用范围标识，非后端相关）                  |
+| `branch`、`request_id`    | 去除（运行时元数据，非统一持久化字段）             |
+
+## 摘要比较策略
+
+框架区分两层摘要正确性：
+
+1. **内容语义一致性** — `summary_text` 在空白符归一化后进行比较。不影响含义的微小格式差异按后端对允许规则处理。
+
+2. **元数据完整性** — `session_id`、`original_event_count`、`compressed_event_count` 必须在各后端间完全匹配。任何偏差均视为无条件失败。框架明确检测三类摘要缺陷：
+   - **摘要丢失** — 摘要存在于后端 A，但后端 B 中缺失。
+   - **摘要覆盖错误** — 摘要存在但 `session_id` 不正确。
+   - **摘要归属错误** — 摘要在底层缓存或存储中被错误地关联到不正确的 session 键。
+
+摘要通过直接注入 `SummarizerSessionManager._summarizer_cache` 的方式生成，绕过 LLM，因为框架测试的是存储一致性而非摘要模型质量。
+
+## 允许差异
+
+并非所有字段级差异都表示缺陷。已知且有文档记录的差异通过 `AllowedDiff` 规则管理：
+
+```
+allowed_diffs = {
+    "inmem_vs_sql": [
+        {"field": "events[*].function_calls[*].args",
+         "reason": "SQL 对 JSON 参数的序列化与 InMemory dict 往返不同"},
+    ],
+    "inmem_vs_redis": [
+        {"field": "events[*].timestamp_precision",
+         "reason": "Redis 将时间戳存储为精度有限的浮点字符串"},
+    ],
+}
+```
+
+每条规则包含后端对、字段路径和原因说明。匹配允许规则的差异不计入失败统计，但仍会在报告中以 `allowed: true` 标记呈现。
+
+## 后端接入方式
+
+| 后端     | 可用性             | 激活方式                                      |
+|----------|--------------------|----------------------------------------------|
+| InMemory | 始终可用           | 直接实例化                                    |
+| SQL      | 始终可用（sqlite）  | `sqlite:///:memory:` — 无外部依赖              |
+| Redis    | 按需启用           | 设置 `TRPC_REDIS_URL` 环境变量；否则跳过        |
+
+轻量模式（InMemory + SQL）无需外部服务即可在 CI 中运行。Redis 集成模式需要设置相应的环境变量，未设置时跳过 Redis 测试对并给出明确提示。
+
+## 差异报告
+
+报告（`session_memory_summary_diff_report.json`）包含：
+
+- **运行元数据**：运行 ID、时间戳、已测试的后端列表。
+- **逐用例结果**：状态（pass/fail/error）、`DiffEntry` 列表。
+- **汇总**：总计/通过/失败数量和误报率。
+
+每个 `DiffEntry` 精确定位：后端对、session ID、事件索引（或摘要 ID）、类别（events/state/memory/summary）、完整的点分隔字段路径以及两个冲突值。
+
+## 框架自身的测试
+
+- `test_replay_normalizer.py` — 每个归一化函数的单元测试。
+- `test_replay_comparator.py` — 差异逻辑的单元测试，包含针对摘要丢失、覆盖错误和归属错误检测的专项测试。
+- `test_replay_report.py` — 报告生成和序列化的单元测试。
+- `test_replay_consistency.py` — 端到端测试，通过后端对运行全部 10 个回放用例，验证正常用例误报率 ≤ 5%，异常用例 100% 检出。

tests/sessions/replay_cases/01_single_turn.json

@@ -0,0 +1,25 @@
+{
+    "case_id": "01_single_turn",
+    "description": "Single turn: user sends a message and agent responds with plain text",
+    "session_setup": {
+        "app_name": "replay_test",
+        "user_id": "test_user",
+        "session_id": "s-001"
+    },
+    "operations": [
+        {
+            "op": "append_event",
+            "author": "user",
+            "text": "Hello, how are you?"
+        },
+        {
+            "op": "append_event",
+            "author": "assistant",
+            "text": "I'm doing well, thank you for asking!"
+        },
+        {
+            "op": "read_back",
+            "expected_event_count": 2
+        }
+    ]
+}

tests/sessions/replay_cases/02_multi_turn.json

@@ -0,0 +1,45 @@
+{
+    "case_id": "02_multi_turn",
+    "description": "Multi-turn conversation: three rounds of user/assistant exchange",
+    "session_setup": {
+        "app_name": "replay_test",
+        "user_id": "test_user",
+        "session_id": "s-002"
+    },
+    "operations": [
+        {
+            "op": "append_event",
+            "author": "user",
+            "text": "What's the weather today?"
+        },
+        {
+            "op": "append_event",
+            "author": "assistant",
+            "text": "It's sunny and 72 degrees."
+        },
+        {
+            "op": "append_event",
+            "author": "user",
+            "text": "Great, I'll go for a walk."
+        },
+        {
+            "op": "append_event",
+            "author": "assistant",
+            "text": "Enjoy your walk! Don't forget sunscreen."
+        },
+        {
+            "op": "append_event",
+            "author": "user",
+            "text": "What about tomorrow?"
+        },
+        {
+            "op": "append_event",
+            "author": "assistant",
+            "text": "Tomorrow will be cloudy with a chance of rain."
+        },
+        {
+            "op": "read_back",
+            "expected_event_count": 6
+        }
+    ]
+}

tests/sessions/replay_cases/03_tool_call.json

@@ -0,0 +1,48 @@
+{
+    "case_id": "03_tool_call",
+    "description": "Tool call conversation: user query triggers a function_call, followed by function_response and final answer",
+    "session_setup": {
+        "app_name": "replay_test",
+        "user_id": "test_user",
+        "session_id": "s-003"
+    },
+    "operations": [
+        {
+            "op": "append_event",
+            "author": "user",
+            "text": "Search for Python async tutorials"
+        },
+        {
+            "op": "append_event",
+            "author": "assistant",
+            "function_call": {
+                "name": "web_search",
+                "args": {
+                    "query": "Python async tutorials"
+                }
+            }
+        },
+        {
+            "op": "append_event",
+            "author": "user",
+            "function_response": {
+                "name": "web_search",
+                "response": {
+                    "results": [
+                        "AsyncIO Tutorial for Beginners",
+                        "Advanced Async Patterns in Python"
+                    ]
+                }
+            }
+        },
+        {
+            "op": "append_event",
+            "author": "assistant",
+            "text": "I found two relevant tutorials: AsyncIO Tutorial for Beginners and Advanced Async Patterns in Python."
+        },
+        {
+            "op": "read_back",
+            "expected_event_count": 4
+        }
+    ]
+}