Background
BEAM evaluates long-term conversational memory at 128K, 500K, 1M, and 10M-token scales across diverse domains and memory abilities. ReMe needs a scalable BEAM harness to characterize memory quality and resource behavior beyond the context sizes covered by smaller conversational benchmarks.
Changes
- Add a BEAM benchmark workflow under
benchmark/beam
- Import the official conversations, questions, annotations, and evaluation assets without committing downloaded or generated data
- Ingest conversations incrementally and preserve turn order, timestamps, domain, scale, and evidence identifiers
- Support representative smoke-test subsets and full 128K, 500K, 1M, and 10M-token tiers
- Reproduce the official evaluation protocol and report results by memory ability, domain, and context scale
- Measure retrieval evidence quality separately from end-to-end answer quality
- Track ingestion time, index size, query latency, model and embedding usage, token usage, and cost
- Make long runs resumable and add configuration documentation, result schemas, and focused fixture-based tests
Background
BEAM evaluates long-term conversational memory at 128K, 500K, 1M, and 10M-token scales across diverse domains and memory abilities. ReMe needs a scalable BEAM harness to characterize memory quality and resource behavior beyond the context sizes covered by smaller conversational benchmarks.
Changes
benchmark/beam