refactor(accuracy): re-home results processor as accuracy accumulator

[ajcasagrande]

Stacked on #1102 (ajc/metrics-accumulator-engine) — retargets to main when that merges.

What moved and why

AccuracyResultsProcessor is re-homed as AccuracyAccumulator (src/aiperf/accuracy/accuracy_results_processor.py -> src/aiperf/accuracy/accuracy_accumulator.py, git rename, 81% similar) and its registration moves from results_processor:accuracy_results to accumulator:accuracy_results with record_types: [metric_records].

With #1102's accumulator engine, legacy results_processor plugins no longer feed ProfileResults.records — RecordsManager summarizes only the registered accumulators (_summarize_metric_record_accumulators), so accuracy summaries were dropped from results on the engine path. This slice plugs accuracy into the engine:

• process_result -> process_record (engine per-record dispatch via _send_record_to_accumulators)
• summarize() gains an optional ctx: SummaryContext | None = None for protocol compatibility (engine's realtime + final paths call summarize() with no args)
• on_dataset_configured keeps working — RecordsManager already duck-calls it on accumulators
• Aggregation logic, emitted tags (accuracy.overall, accuracy.task.*, accuracy.unparsed*), and the self-disable (PostProcessorDisabled when accuracy mode is off) are unchanged

Also carried: docstring renames in accuracy_dataset_loader.py, doc reference updates in docs/accuracy/, and regenerated plugin enum docstrings (src/aiperf/plugin/enums.py).

Tests

• Renamed test_accuracy_results_processor_summarize.py -> test_accuracy_accumulator_summarize.py (mechanical)
• Updated test_accuracy_record_processor.py for the new names
• New test_accuracy_accumulator_protocol_path.py: drives the accumulator through the registered plugin path — registry class resolution, the exact load_accumulators constructor kwargs, record_types metadata routing, and summarize() via generate_realtime_metrics (the engine helper) — so seam drift fails tests instead of only crashing at runtime

Verification

• uv run pytest -n auto tests/unit/accuracy/ tests/unit/records/ tests/unit/metrics/ — 1270 passed, 11 skipped
• uv run pytest --collect-only -q tests/unit/ — 14288 collected, no errors
• make check-ruff-baselined (119 baselined, 0 new), make check-ergonomics (61 baselined, 0 new)
• make validate-plugin-schemas (235 plugins OK), make generate-all-plugin-files — zero drift

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Accuracy processing now runs as part of the accumulator flow, with support for record-level handling and summary reporting.
  • The accuracy plugin is now registered in the appropriate pipeline for metric records.

• Documentation

  • Updated accuracy docs and examples to match the current end-to-end setup and terminology.

• Tests

  • Added and updated tests for plugin registration, routing, disabling behavior, and accuracy summaries.

[github-actions]

Try out this PR

Quick install:

pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@171e52a264351502a85d990f53f4450fc40db378

Recommended with virtual environment (using uv):

uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@171e52a264351502a85d990f53f4450fc40db378

---

Last updated for commit: 171e52a • Browse code

[github-actions]

Fern Docs Preview: https://nvidia-preview-a75948f8-f019-4318-9af6-60dccd535b22.docs.buildwithfern.com/aiperf/dev

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[coderabbitai]

Walkthrough

The accuracy post-processor class AccuracyResultsProcessor is renamed to AccuracyAccumulator, with process_result renamed to process_record and summarize gaining an optional ctx parameter. Its plugin registration moves from results_processor to accumulator. Tests and docs are updated accordingly.

Changes

AccuracyAccumulator rename and re-registration

Layer / File(s)	Summary
Core implementation rename src/aiperf/accuracy/accuracy_accumulator.py, src/aiperf/dataset/loader/accuracy_dataset_loader.py	Class renamed to AccuracyAccumulator, process_result renamed to process_record, error messages updated, and summarize accepts an optional `ctx: SummaryContext
Plugin registry migration src/aiperf/plugin/plugins.yaml, src/aiperf/plugin/enums.py	The accuracy_results plugin entry moves from results_processor to accumulator category using AccuracyAccumulator; enum docstring examples updated.
Unit tests tests/unit/accuracy/test_accuracy_accumulator_summarize.py, tests/unit/accuracy/test_accuracy_record_processor.py, tests/unit/accuracy/test_accuracy_accumulator_protocol_path.py	Existing tests updated to use AccuracyAccumulator/process_record; new test module validates plugin resolution, record-type routing, disable behavior, and realtime summarization.
Documentation docs/accuracy/accuracy-benchmarking.md, docs/accuracy/accuracy_stubs.md	Architecture diagrams, plugin tables, implementation counts, and file references updated to reference AccuracyAccumulator.

Estimated code review effort: 2 (Simple) | ~15 minutes

Poem

A rabbit renamed with a hop and a cheer,
Processor to Accumulator, crystal clear,
Records now flow through process_record's door,
Summaries counted, correct to the core,
Tests all rewritten, docs realigned —
🐇 a tidy little burrow, freshly designed.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: re-homing the accuracy results processor as an accuracy accumulator.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

tests/unit/accuracy/test_accuracy_record_processor.py (1)

8-9: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

File name no longer matches its content.

This file exclusively tests AccuracyAccumulator now, but is still named test_accuracy_record_processor.py, while a distinct AccuracyRecordProcessor class exists elsewhere in the codebase. This naming collision risks confusing future contributors about which class is under test.

♻️ Suggested rename

git mv tests/unit/accuracy/test_accuracy_record_processor.py tests/unit/accuracy/test_accuracy_accumulator.py

Also applies to: 48-49, 247-248, 277-278

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unit/accuracy/test_accuracy_record_processor.py` around lines 8 - 9,
The test module name no longer matches what it covers: it now tests
AccuracyAccumulator only, but is still named as if it targets
AccuracyRecordProcessor. Rename the test file to match its actual subject and
update the imports/usages in the test module so AccuracyAccumulator is the only
referenced class; keep AccuracyRecordProcessor references out of this file since
that class is tested elsewhere.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/unit/accuracy/test_accuracy_record_processor.py`:
- Around line 8-9: The test module name no longer matches what it covers: it now
tests AccuracyAccumulator only, but is still named as if it targets
AccuracyRecordProcessor. Rename the test file to match its actual subject and
update the imports/usages in the test module so AccuracyAccumulator is the only
referenced class; keep AccuracyRecordProcessor references out of this file since
that class is tested elsewhere.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 89c3aa76-5b98-40be-a2e8-a03c5f5b428a

📥 Commits

Reviewing files that changed from the base of the PR and between d41bd50 and 171e52a.

📒 Files selected for processing (9)

• docs/accuracy/accuracy-benchmarking.md
• docs/accuracy/accuracy_stubs.md
• src/aiperf/accuracy/accuracy_accumulator.py
• src/aiperf/dataset/loader/accuracy_dataset_loader.py
• src/aiperf/plugin/enums.py
• src/aiperf/plugin/plugins.yaml
• tests/unit/accuracy/test_accuracy_accumulator_protocol_path.py
• tests/unit/accuracy/test_accuracy_accumulator_summarize.py
• tests/unit/accuracy/test_accuracy_record_processor.py

[ajcasagrande]

I found one correctness issue that should be fixed before merge: the new AccuracyAccumulator is wired into the accumulator pipeline but still behaves like the old all-run results processor, so warmup records are included in final profiling accuracy totals whenever warmup is enabled. The source path confirms the accumulator receives all metric records and stores only global counters, and the runtime reproduction with the mock server produced accuracy_results.csv totals of 2 for a run with one warmup request and one profiling request.

Everything else in the re-home looks directionally sound: the plugin registration moved to accumulator:accuracy_results, the record-type metadata routes it through the metric-record accumulator path, and disabled-accuracy construction still raises PostProcessorDisabled as expected. Fix order: first make accuracy summaries phase/window-aware and add a warmup+profiling regression test through the accumulator/RecordsManager path; then rerun the targeted accuracy tests and CLI repro.

[ajcasagrande]

This loses the phase scoping that the accumulator pipeline relies on. After moving accuracy aggregation from the legacy results-processor path into accumulator:accuracy_results, every warmup and profiling MetricRecordsData is now sent through process_record, but the accumulator keeps only one set of counters and explicitly ignores the SummaryContext. The final RecordsManager path asks for profiling and warmup summaries separately, so an accuracy run with --warmup-request-count 1 --request-count 1 exports Total 2 in accuracy_results.csv for the profiling result. I reproduced that with the real CLI and in-repo mock server, and a direct accumulator repro with one wrong warmup record plus one correct profiling record returns the same mixed count=2/current=0.5 for both SummaryContext(PROFILING) and SummaryContext(WARMUP). Please store enough per-record phase/window data and honor the summary/export context (and make sure the RecordsManager path passes it here) before emitting accuracy MetricResults.