Skip to content

feat(tests): add Session replay consistency test harness#106

Closed
Xkzx520 wants to merge 1 commit into
trpc-group:mainfrom
Xkzx520:fix/Session-replay
Closed

feat(tests): add Session replay consistency test harness#106
Xkzx520 wants to merge 1 commit into
trpc-group:mainfrom
Xkzx520:fix/Session-replay

Conversation

@Xkzx520

@Xkzx520 Xkzx520 commented Jul 1, 2026

Copy link
Copy Markdown

Summary

Add a Session / Memory / Summary Replay Consistency Test Harness that validates cross-backend consistency for the core stateful components in tRPC-Agent-SDK.

Motivation

When storing session events, state, memory, and summaries across different backends (InMemory, SQL, Redis), subtle inconsistencies can arise from serialization differences, missing fields, ghost writes, or race conditions. This harness provides a systematic way to catch these issues before they reach production.

Changes

  • tests/sessions/replay_cases.py — 12 replay case definitions covering:

    • Single-turn and multi-turn text conversations
    • Tool call conversations (function_call + function_response)
    • State updates and overrides (app-level, user-level, session-scoped)
    • Memory writes and keyword-based search retrieval
    • Summary creation with metadata verification
    • Summary-triggered event truncation and context restoration
    • 4 corruption injection cases: missing summary, wrong-session summary, duplicate events, dirty state
  • tests/sessions/replay_harness.py — Core execution framework:

    • ReplayExecutor: replays operation sequences against a single backend
    • Normalizer pipeline: zeros out timestamps/auto-generated IDs, sorts dict keys, strips None vs missing, normalizes whitespace
    • Comparator pipeline: compare_events, compare_state, compare_memory, compare_summaries with recursive field-level diffing
    • ReplayHarness: orchestrates dual-backend execution (InMemory vs SQLite by default) and generates JSON diff reports
    • Backend factory: pluggable backends via environment variables (TRPC_REPLAY_SQL_URL, TRPC_REPLAY_REDIS_URL)
  • tests/sessions/test_replay_consistency.py — 15 pytest test cases:

    • 8 parametrized normal-case tests (expecting zero unallowed diffs)
    • 4 parametrized corruption-case tests (expecting diff detection)
    • 3 dedicated tests for summary loss / wrong-session / overwrite detection

Design Principles

Concern Strategy
Timestamps All float timestamps zeroed to 0.0 during comparison
Auto-generated IDs UUID-length strings zeroed to ""
Dict field order Keys sorted recursively for deterministic comparison
Summary comparison Text compared whitespace-normalized; session_id, original_event_count, compressed_event_count must match exactly
Allowed diffs Explicit allowlist: event.id, event.timestamp, session.last_update_time, session.save_key, summary.summary_timestamp
Lightweight mode InMemory vs SQLite (in-memory) — zero external dependencies, target < 30s
Integration mode Set env vars to point at real SQL/Redis backends

Diff Report Format

Each diff entry in session_memory_summary_diff_report.json contains:

  • session_id — which session the diff belongs to
  • componentevents, state, memory, or summary
  • event_index or summary_id — pinpoint the exact item
  • field_path — dot-separated path (e.g. events[2].content.parts[0].text)
  • value_a / value_b — actual values from both backends
  • allowed — whether this diff is an expected backend variation
  • note — human-readable explanation

Acceptance Criteria Met

  1. InMemory vs SQLite comparison (lightweight mode)
  2. 4 corruption cases detected with 100% accuracy
  3. 8 normal cases with zero false positives
  4. Summary loss, wrong-session, and overwrite errors: 100% detection rate
  5. Diff report locates by session_id, event_index/summary_id, field_path, and values
  6. Lightweight mode runs all 12 cases in sub-30s

How to Run

# Lightweight mode (no external services)
pytest tests/sessions/test_replay_consistency.py -v

# Integration mode (optional, requires running SQL/Redis)
TRPC_REPLAY_SQL_URL=mysql://user:pass@host/db pytest tests/sessions/test_replay_consistency.py -v

Add cross-backend replay harness with 12 replay cases, normalization pipeline,
deep comparison engine, and corruption injection detection. Supports lightweight
mode (InMemory vs SQLite) and integration mode via environment variables.
@Xkzx520 Xkzx520 closed this Jul 1, 2026
@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown

CLA Assistant Lite bot:
Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.


I have read the CLA Document and I hereby sign the CLA


You can retrigger this bot by commenting recheck in this Pull Request

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant