Skip to content

Implement independent multi-annotator labeling and adjudication #46

Description

@alexschmander

Problem

Labello cannot operationally collect independent labels, calculate agreement, or route disagreements through an authorized adjudication workflow.

Current behavior

Agreement/adjudication shapes and roles exist, but production Adjudicate assignment/UI are disabled and submissions are not isolated as independent candidates.

Desired behavior

Implement the complete independent-labeling, agreement, automatic acceptance, disagreement routing, and adjudication lifecycle.

Acceptance criteria

  • Keep annotator submissions isolated until the configured independent-label count is reached.
  • Compute configured IoU/keypoint-distance agreement deterministically with versioned policy.
  • Automatically finalize agreement and route disagreement to an authorized adjudicator.
  • Enable Adjudicate availability, claiming, UI decisions, replay, statistics, and administration.
  • Preserve annotator independence, reviewer/adjudicator separation, assignment race safety, and provenance across restart/offline/version migration.

Scope

Cross-cutting domain, storage, API/client, UI, configuration, persistence, and statistics capability.

Implementation notes

Do not enable scaffolded routes/UI until authorization, replay, assignment, and durable data contracts are complete.

Tracking source

docs/tracking/feature-requests.md:19

Independent multi-annotator labeling and adjudication.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: domainPure domain policy, events, replay, and validationenhancementNew feature or requestreliabilityRecovery, durability, failure handling, and operational robustnessworkflow: annotationAnnotation assignment and editing workflowworkflow: reviewApproval review and reviewer correction workflow

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions