Skip to content

Add active learning feedback loop to PhEye and Phileas #2

Description

@jzonthemtn

Summary

Arbiter captures human review decisions (accept, override, exempt) but these corrections are not fed back to PhEye or Phileas to improve detection accuracy over time.

Proposal

Close the feedback loop:

  • Export review decisions in training-data format (CoNLL, spaCy)
  • Identify high-value corrections (false positives, overrides) for retraining
  • Provide an API or export mechanism for PhEye's fine-tuning pipeline
  • Track the impact of human feedback on model accuracy over retraining cycles

This is the single highest-leverage cross-project integration for improving the entire toolkit.

Acceptance Criteria

  • Reviewer decisions (accept, override, exempt) from Arbiter are exported in a format consumable by PhEye and Phileas
  • Feedback improves detection accuracy over time (measurable via Philter Scope benchmarks)
  • Privacy preserved: no raw PII in the feedback data, only entity spans and decisions
  • Process documented end-to-end
  • Tests cover feedback export and ingestion

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions