Skip to content

Auto-mode permission classifier hygiene (effort, schema, fail-open, memoization, usage accounting) #312

Description

@jbellis

The Auto-mode permission classifier has measured pathologies (profiled 2026-07-28 across 25 AWS benchmark attempts, see dossier/followup-tickets-2026-07-28.md in the anvil-checkpoints branch):

  • Inherits the session's reasoning effort (tool_loop.rs ~4452): a trivial allow/deny call at xhigh costs ~7s. Force a low effort for classifier calls.
  • prefer_json_object output (~4564) at high effort yields a 1.2x empty-completion retry storm. Use strict json_schema.
  • Unavailable maps to Reject (~4208-4223): 19% of decisions denied legitimate tool calls and burned a main-loop turn each. Fail open with a visible notice (or prompt).
  • No memoization: repeated edits to the same file re-classify from scratch. Cache on (tool, normalized input).
  • Retry-attempt and unavailable-response usage bypass session usage accounting (the permission_classifier trace record added in 3d3d492 captures true sums; accounting should too).

Benchmark runs now bypass the classifier entirely via BROKK_ACP_PERMISSION_MODE; these items are for interactive Auto-mode users, who still pay all of the above.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions