Skip to content

Implement consolidated summary and failed example reporting in orchestrator #26

Description

@vovka

Background

Currently, the orchestrator prints only a dummy summary at the end of parallel test runs. For full usability, it must print a consolidated summary, including failed examples, and test statistics across all processes.

Requirements

  1. Formatter Data Collection

    • Each formatter (test process) must collect:
      • List of failed examples with:
        • Description
        • File/line location
        • Error message (backtrace optional)
      • Example counters:
        • Total examples
        • Failed examples.
        • Pending examples
      • Elapsed time for its process (optional but useful for reporting)
      • If there is already exist rspec built-in mechanism for this, use it, do not invent new.
  2. Summary Transmission

    • At the end of the test suite (on dump_summary or close), each formatter must send a new IPC message to the orchestrator.
    • Define a new IPC message type (e.g., summary).
    • Payload should include all items listed above.
    • Pending example details should NOT be sent—only the count.
  3. Orchestrator Summary Rendering

    • The orchestrator must collect all summary messages from all processes.
    • Once all are received, it should:
      • Merge all failures into a single list.
      • Sum total, failed, and pending example counts across all processes.
      • Sum or list all process run times (if available).
    • Print a consolidated summary at the end, including:
      • List of failed examples, with details. (Should look like standard rspec failures output - test name, error, expected/got, diff (3 previous items colorized), backtrace.)
      • Statistics: N examples, M failures, K pending.
      • Total wall clock time.
      • Sum of process times.
    • Pending examples: only print the total count (no details).
  4. Robustness

    • The orchestrator should robustly handle missing or delayed summaries (e.g., with a timeout or best-effort fallback), to avoid hanging indefinitely.
    • Prefer IPC for summary transmission unless unworkable; do not use files except as a fallback.
  5. Simplicity

    • Prefer the simplest working solution. Do not introduce unnecessary abstractions.

Acceptance Criteria

  • After all test processes finish, the orchestrator prints a consolidated summary and failure listing, as described above.
  • No pending test details are printed (only the count).
  • Solution uses IPC for summary sharing.
  • If a process crashes or fails to report, orchestrator prints the summary with what it received, and logs a warning.

This issue is based on design analysis and requirements clarification. If anything is unclear or you see edge cases not covered, please add questions or comments before implementation.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions