Skip to content

Add multi-reviewer agreement and adjudication workflow #3

Description

@jzonthemtn

Summary

Arbiter has no inter-annotator agreement metrics or workflow for resolving disagreements between reviewers. In regulated environments, multiple reviewers may need to review the same document.

Proposal

Add multi-reviewer support:

  • Assign the same document to multiple reviewers
  • Calculate inter-annotator agreement (Cohen's kappa or similar)
  • Adjudication workflow for resolving disagreements
  • Track agreement metrics over time for quality assurance

Acceptance Criteria

  • Multiple reviewers can be assigned to the same document
  • Agreement metrics computed (inter-annotator agreement / Cohen's kappa)
  • Disagreements flagged for adjudication by a senior reviewer
  • Adjudication decision is final and logged in the audit trail
  • Tests cover agreement calculation and adjudication workflow

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions