skill re-audit: decide whether to eval the 54 held behavior-changing rows #15516

Activity

1. github-actions

   added

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Nov 12, 2026
2. github-actions commented on Nov 12, 2026

   

   github-actionsBot

   on Nov 12, 2026 – with GitHub Actions

   Contributor

   More actions

   Size: Large
   Complexity: Medium
   Uncertainty: Known
   Flags: intent
   Blocked by: none

   Understood: 54 held behavior-changing skill rows have no eval coverage, and the issue asks the owner whether to evaluate them. No duplicate or merged fix found. Related work is #15469 and #15468. The issue says its outcome is a decision, so these questions are open:

   1. Scope: run all 54 rows, a subset by plugin, or none? RECOMMENDED: run all 54. The issue projects about 20 new cases and about 7 cuts shipping, and the pilot's rates are the only basis. Eval runs bill the prepaid API keys, so this is not a new charge.
   2. Margin: what non-inferiority margin applies under [D10] on Opus 5.5 and Fable 5.2? RECOMMENDED: reuse the pilot's margin from docs(harness-ops): ship eval-gated held rows and three stale-fact fixes #15469. That keeps results comparable.

   To answer, post a comment that starts with this line:

   **Decision recorded.** <answer>
   

   _{Automated triage. A maintainer reviews it.}
3. github-actions

   added

   needs-owner-decisionAwaiting a human or maintainer judgment call.Awaiting a human or maintainer judgment call.

   on Nov 12, 2026
4. github-actions

   set

   Priority

   to

   medium

   on Nov 12, 2026
5. github-actions

   added the

   Task

   issue type on Nov 12, 2026
6. melodic-automation-lanes commented on Nov 12, 2026

   

   melodic-automation-lanesBot

   on Nov 12, 2026

   Contributor

   More actions

   The triage verdict left this issue at needs-owner-decision:

   • flagged: intent
   • held by needs-owner-decision
   • no lanes App verdict comment shows whether an earlier verdict was flagged

   Reason: intent
7. melodic-automation-lanes

   set

   Work status

   to

   needs-owner-decision

   on Nov 12, 2026
8. melodic-automation-lanes commented on Nov 12, 2026

   

   melodic-automation-lanesBot

   on Nov 12, 2026

   Contributor

   More actions

   Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

   Intended outcome: Run the eval suite on the 54 held rows (all, a subset, or none) under the D10 non-inferiority rule on Opus 5.5 and Fable 5.2, then ship only the cuts that pass.

   Related work

   • docs(planning,bugs): restore rule reasons the re-audit pointers dropped #15288 (related): merged PR restoring rule reasons for planning and bugs re-audit rows; may overlap bugs-4 and planning rows
   • docs(architecture,event-storming): point re-audit findings at live sources #15234 (related): merged PR pointing architecture re-audit findings at live sources; may affect architecture-4 and architecture-5

   Open questions

   1. Scope: run evals for all 54 rows, a subset by plugin, or none?
      RECOMMENDED: Run all 54 rows in one pass, since about 20 new cases cost about $142 and give one comparable result set. Rows with no source and no reason can be dropped from the run first and left unchanged. Basis: judgment: the issue's projected cost of about $142 and about 3.4 hours, in the issue body's cost section
   2. Margin: what non-inferiority margin applies under D10 on both Opus 5.5 and Fable 5.2?
      RECOMMENDED: Reuse the margin the #​15469 pilot used so the results stay comparable. This run could not read that margin, so the owner must confirm it. Basis: #​15469
   3. Should rows that only widen a delegation condition (inline path for small scope) be evaluated, or left as written because the cut is optional?
      RECOMMENDED: Evaluate them, since D10 requires measured non-inferiority for any change to when a skill delegates. Rows that propose no change (work-items-22) should be dropped. Basis: judgment: the issue body states that D10 gates cuts and that work-items-22 proposes no change
   4. Do rows already touched by merged restore-reason PRs (#​15288, #​15234) still need evals, or are they now resolved?
      RECOMMENDED: Recheck the bugs, planning and architecture rows against main before running, and drop any that those PRs already settled. Basis: #​15288

   To answer, post a comment that starts with this line:

   **Decision recorded.** <answer>
   

   _{Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.}