Adding support for Temporal Grounding and searchable summaries

[aayush3011]

Summary

Two improvements:

• Make generated summaries retrievable via vector search and blendable into search_cosmos
• A v3 extraction-prompt pass that sharpens attribution, temporal grounding, and completeness. Retrieval
  additions are opt-in; the prompt change is behind a version bump (v3) and takes effect on re-memorize.

Changes

Searchable summaries

Previously the summaries container had no vector index, so generated
thread_summary / user_summary docs were not reachable by search.

• The summaries container is now provisioned with a vector + full-text index
  (keeping the existing (user_id, thread_id, version) composite for point
  lookups).
• New search_summaries(query, user_id, thread_id=None) (sync + async, store +
  client) vector-searches across all of a user's summary docs at once - the
  single user_summary and every per-thread thread_summary - so a holistic,
  session-scoped, or topic-scoped summary question all resolve with one query.
• search_cosmos(..., include_summaries=True, summary_top_k=...) blends matching
  summaries into results (best-effort). With include_turns also on, the result
  order is facts → summaries → raw turns, so summaries/turns never jump the
  relevance-ranked facts or evict them under context truncation.

Extraction prompt (v2 → v3)

• Named attribution: use a person's actual name when the conversation gives
  one (the user if named, and always third parties) instead of collapsing
  everyone into "the user" - essential for multi-speaker transcripts so two
  speakers don't blur into one.
• Temporal grounding - "two clocks": an explicit event-time vs
  processing-time guard, so relative expressions ("3 weeks ago") resolve only
  against the turn's timestamp, never against the current clock; plus
  relative→absolute (never absolute→vague) and preserve exact durations/counts.
• Meaning/polarity: don't flip the direction of a statement ("moved off
  Datadog" ≠ "uses Datadog").
• Content, not the act of sharing: extract the facts inside a shared
  document/spec/dataset, not "the user shared a doc".
• Exhaustive-scan self-check: re-scan the whole transcript (middle and end),
  cover every distinct topic - without the volume quotas that would inflate count.
• Two new few-shot examples (document→content, multi-topic→per-dimension).

Minor

• created_at normalization drops a redundant Z→offset replace
  (datetime.fromisoformat parses Z natively on the supported Python range).

Tests

Store + client coverage for search_summaries scoping, include_summaries
blending + facts→summaries→turns ordering, the summaries container vector policy,
and the prompt version bump. Full unit suite green; ruff clean.

Benchmark Results

Beam 1M

LongMemEval

LoCoMo

[Copilot]

Pull request overview

This PR expands the Cosmos-backed memory system to support semantically searchable summaries and improves fact extraction guidance around event-time vs processing-time temporal grounding.

Changes:

• Add search_summaries to sync + async MemoryStore and expose it via both Cosmos memory clients.
• Make the summaries container carry vector + full-text indexing policies so summaries can be queried via semantic search.
• Update extract_memories.prompty to v3 with stronger temporal grounding and attribution rules; update tests accordingly.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/unit/test_cosmos_memory_client.py	Updates expectations for summaries container policies; adds sync tests for summaries inclusion and ordering.
tests/unit/store/test_memory_store.py	Adds unit coverage for MemoryStore.search_summaries query shape and validation.
tests/unit/services/test_prompty_loader.py	Bumps expected extract_memories.prompty version to v3.
tests/unit/aio/test_cosmos_memory_client.py	Adds async tests for including summaries and ordering with turns.
azure/cosmos/agent_memory/store/memory_store.py	Implements sync MemoryStore.search_summaries using shared search helpers.
azure/cosmos/agent_memory/prompts/extract_memories.prompty	Updates extraction prompt to v3 with event-time grounding and richer formatting rules/examples.
azure/cosmos/agent_memory/cosmos_memory_client.py	Adds summaries indexing policies on container creation; adds include_summaries blending and search_summaries API.
azure/cosmos/agent_memory/aio/store/memory_store.py	Implements async search_summaries.
azure/cosmos/agent_memory/aio/cosmos_memory_client.py	Adds summaries indexing policies on container creation; adds async include_summaries blending and search_summaries API.
azure/cosmos/agent_memory/_utils.py	Simplifies ISO parsing in normalize_created_at_iso.

Comments suppressed due to low confidence (1)

azure/cosmos/agent_memory/cosmos_memory_client.py:742

• This warning message says "returning memories only", but the function no longer returns early on turn-search failure and may still return summaries (and always returns whatever memory hits were found). Update the log message to match the behavior.

            except Exception as exc:  # noqa: BLE001
                logger.warning("search_cosmos: include_turns turn search failed (%s); returning memories only", exc)
                turns = []

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.