Weka loader caps before context filtering and silently wraps a too-small conversation pool

[PeaBrane]

Summary

A customer router comparison used the Weka HF dataset with --max-context-length 131072 and --concurrency 100, reasonably expecting the default 100 dataset entries to provide roughly 100 distinct eligible source conversations.

Instead, AIPerf:

1. selected the first 100 Weka rows because --num-dataset-entries defaults to 100;
2. applied the max-context filter later, which rejected 85 of those rows;
3. retained only 15 sampleable root conversations; and
4. used the preferred sequential sampler, which silently wrapped those 15 roots into 100 runtime sessions.

That produced 6–7 concurrent exact clones of each source conversation, each with a distinct runtime/session ID. This materially changed prefix-cache locality and router behavior, making the result unsuitable as a representative comparison.

This was observed on an AgentX/Weka build based on 8473e15, reporting AIPerf 0.8.0.

Reproduction shape

The relevant arguments were:

aiperf profile \
  --public-dataset weka_hf \
  --hf-weka-repo semianalysisai/cc-traces-weka-with-subagents-051926 \
  --max-context-length 131072 \
  --concurrency 100 \
  --benchmark-duration 600 \
  --request-count 50000 \
  --random-seed 20260702 \
  --no-fixed-schedule \
  --ignore-trace-delays

There was no explicit --num-dataset-entries/--num-prompts or sampling-strategy override.

The surprising interaction is:

• SemiAnalysisCCTracesWekaLoader._validate_rows() caps the raw dataset before conversion/filtering.
• WekaTraceLoader applies max_context_length afterward.
• SequentialSampler wraps indefinitely when the reduced pool is exhausted.

In the observed run, 100 runtime sessions came from only 15 source roots. Trace analysis found 394 of 465 exact-sequence follow-ups crossing runtime-session boundaries, with 348 of those arriving within five seconds. A clone-free control removed the large apparent cache-locality benefit.

Expected behavior

For trace-backed datasets, requesting 100 dataset entries should yield up to 100 distinct eligible source traces after configured filters are applied. If the corpus cannot supply that many, AIPerf should fail clearly or require an explicit opt-in to reuse/wrap the reduced pool.

At minimum, AIPerf should not silently run concurrency 100 over 15 eligible roots while presenting the run as a 100-entry dataset.

Suggested changes

1. Scan/filter Weka rows until the requested number of distinct eligible source traces has been collected, or the corpus is exhausted; apply the cap after filtering rather than before it.
2. When the unique eligible pool is smaller than the requested concurrent sessions, fail by default or require an explicit option such as --allow-dataset-wrap.
3. Log/report total rows, rows scanned, rows rejected by each filter, eligible unique source roots, runtime sessions, wrap count, and effective duplication factor.
4. Add regression coverage for cap-before-filter and for silent sequential wrap below concurrency.
5. Clarify that --request-count controls requests, not the number of source traces loaded.

The current behavior makes an understandable benchmark configuration easy to misuse and can substantially bias cache-aware routing comparisons.

[ajcasagrande]

I believe 1. selected the first 100 Weka rows because --num-dataset-entries defaults to 100; is already fixed in agentX v1.0 Pr #1084