Skip to content

benchmark: add BEAM evaluation #355

Description

@jinliyl

Background

BEAM evaluates long-term conversational memory at 128K, 500K, 1M, and 10M-token scales across diverse domains and memory abilities. ReMe needs a scalable BEAM harness to characterize memory quality and resource behavior beyond the context sizes covered by smaller conversational benchmarks.

Changes

  • Add a BEAM benchmark workflow under benchmark/beam
  • Import the official conversations, questions, annotations, and evaluation assets without committing downloaded or generated data
  • Ingest conversations incrementally and preserve turn order, timestamps, domain, scale, and evidence identifiers
  • Support representative smoke-test subsets and full 128K, 500K, 1M, and 10M-token tiers
  • Reproduce the official evaluation protocol and report results by memory ability, domain, and context scale
  • Measure retrieval evidence quality separately from end-to-end answer quality
  • Track ingestion time, index size, query latency, model and embedding usage, token usage, and cost
  • Make long runs resumable and add configuration documentation, result schemas, and focused fixture-based tests

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

Status
In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions