Skip to content

spec(github-scan): auto-revert github-scan:humannew on new human comment #358

Description

@serenakeyitan

Problem

The classifier treats github-scan:human as a one-way trapdoor: once an item is labeled human, the daemon never re-evaluates, even if the human comments back with a decision.

This breaks the natural handoff flow:

  1. Agent escalates: "I need a human to decide X" → labels github-scan:human.
  2. Human reads, replies in the issue with the decision (e.g. "go ahead with option A").
  3. Human expects the agent to read the decision and continue working.

Step 3 silently fails today. The only way out of human is for the human to also manually strip the github-scan:human label — two actions for one logical handoff. People forget, so agents silently sit idle on resolved escalations.

Concrete example

first-tree-website#10 — the human commented @serenakeyitan make it done. The daemon polled, saw the new activity, but classified the item human and skipped it because the label was still set.

Proposed fix (Option A: auto-revert on new human comment, with guard)

When the daemon polls an item classified human and observes a new human comment posted after the label was applied:

  1. Strip github-scan:human from the item.
  2. Item re-classifies as new (default for OPEN items with no github-scan:* label).
  3. Daemon picks it up on the next dispatch cycle.

Guard rails (avoid spurious re-fires)

  • Comment author MUST NOT be the agent itself (the daemon's identity).
  • Comment body length > 20 chars (filter out 👍, ok, thx-only acks).
  • Reactions alone do NOT count as a comment.
  • The comment's created_at MUST be strictly after the label's created_at (or the label-event timestamp from the issue timeline).

Out of scope for this issue

  • A separate /handoff magic phrase or bot-mention trigger (Option B). The auto-revert covers ~95% of the natural flow; explicit triggers can be added later if Option A produces false positives in practice.
  • Tray-driven handoff (clicking a row in the menu-bar tray to also strip the label). Tracked separately once apps/tray-mac lands.

Why this matters now

The friction is invisible while only a handful of human-labeled items exist and the same person who set the label is also acting on them. As soon as the menu-bar tray ships and the queue becomes glanceable, more human items will accumulate and the "reply in GitHub vs. also remember to strip the label" friction will compound — exactly the wrong UX as we're promoting github scan.

Acceptance criteria

  • Daemon auto-strips github-scan:human when guards pass.
  • Item re-classifies as new and is dispatched on the next cycle.
  • Unit tests cover: (a) own-comment ignored, (b) short ack ignored, (c) reaction ignored, (d) genuine reply triggers revert, (e) pre-label comment does NOT trigger revert.
  • Spec doc updated to document the human → new transition.

cc @bingran-you

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions