-
Notifications
You must be signed in to change notification settings - Fork 132
Add LLM issue auto-triage: type and high-severity labelers #584
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+110
−0
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| instructions: | | ||
| This labeler decides one thing: whether the issue is high severity. Almost | ||
| all issues are NOT high severity — return an empty list unless the issue | ||
| clearly and credibly describes one of the qualifying situations below. | ||
| A noisy severity label trains maintainers to ignore it, so precision | ||
| matters far more than recall. | ||
|
|
||
| Qualifies as high-severity ONLY when the issue describes, with concrete | ||
| evidence (error output, versions, reproduction, or a specific code path): | ||
| - A production outage or agents failing to run at all because of the SDK. | ||
| - Data loss or data corruption caused by the SDK (e.g. memory records | ||
| lost or silently dropped). | ||
| - A security vulnerability or credential/secret exposure in the SDK. | ||
| - A crash or hard failure in a core code path with no workaround. | ||
| - A regression: behavior that worked in a previous released version of | ||
| bedrock-agentcore and is broken in a newer release. | ||
|
|
||
| In addition to matching a category above, the failure must affect users in | ||
| production or block adoption of a release. Judge blast radius: would a | ||
| typical production user of the SDK hit this in a core path (importing the | ||
| package, running an agent, storing/retrieving memory)? A failure that only | ||
| occurs in a specific optional integration, edge-case code path, or unusual | ||
| combination of features does not qualify, even with a crash and a solid | ||
| reproduction — those are ordinary bugs. In particular: a crash or | ||
| validation error inside a third-party framework integration (e.g. the | ||
| Strands session manager) that requires a specific feature combination to | ||
| trigger is an ordinary bug, regardless of how detailed the report is. | ||
| Report quality is not severity. | ||
|
|
||
| Does NOT qualify, even if the author says it is urgent or critical: | ||
| - Feature requests and enhancements of any kind. | ||
| - Usage questions, configuration mistakes, or environment problems. | ||
| - Bugs with a documented or obvious workaround. | ||
| - Vague reports with no reproduction, versions, or error output. | ||
| - Issues in other AgentCore components (CLI, service, console) rather | ||
| than this Python SDK. | ||
| - Performance complaints without evidence of a regression or outage. | ||
|
|
||
| When in doubt, return an empty list. | ||
|
|
||
| labels: | ||
| high-severity: "Production outage, data loss, security vulnerability, crash with no workaround, or regression in a released version — with concrete evidence." |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| instructions: | | ||
| Classify the issue by what the author is asking for, not by tone. | ||
| - "bug" requires a claim that existing SDK behavior is broken or incorrect. | ||
| - "enhancement" covers new features, API additions, and improvements to | ||
| existing behavior that is working as documented. | ||
| - "question" covers usage help, clarification requests, and "how do I" | ||
| issues, even when phrased as a problem report. | ||
| If the issue is a bug report that also proposes a fix, label it "bug". | ||
|
|
||
| labels: | ||
| bug: "Existing SDK behavior is broken, incorrect, or crashes." | ||
| enhancement: "Request for a new feature, API addition, or improvement to working behavior." | ||
| question: "Usage question, clarification request, or how-do-I issue." |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| name: Issue Auto Triage | ||
|
|
||
| # Classifies new issues with an LLM (Bedrock via strands-agents/devtools | ||
| # issue-labeler) and applies labels from a hardcoded allowlist. The model has | ||
| # no tools, no shell, and no GitHub access — it returns a structured object | ||
| # whose label field is an enum built from the config allowlist, so the worst | ||
| # a prompt injection in an issue body can achieve is a mislabel. | ||
| # | ||
| # Labeler configs live in .github/labelers/. Each job is an independent | ||
| # classification concern and they run in parallel. | ||
|
|
||
| on: | ||
| issues: | ||
| types: [opened] | ||
|
|
||
| permissions: | ||
| issues: write | ||
| id-token: write | ||
| contents: read | ||
|
|
||
| jobs: | ||
| label-type: | ||
| name: "Label: Type" | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 2 | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| with: | ||
| sparse-checkout: .github/labelers | ||
| sparse-checkout-cone-mode: false | ||
| # strands-agents/devtools/issue-labeler pinned to a reviewed commit | ||
| # (main as of 2026-07-17). Bump deliberately, not automatically. | ||
| - uses: strands-agents/devtools/issue-labeler@5e50593798905abfbb31cc709f5ac58a2992b03a | ||
| with: | ||
| aws_role_arn: ${{ secrets.BEDROCK_SECURITY_REVIEW_ROLE_ARN }} | ||
| config_path: '.github/labelers/type.yml' | ||
| max_labels: '1' | ||
|
|
||
| label-severity: | ||
| name: "Label: Severity" | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 2 | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| with: | ||
| sparse-checkout: .github/labelers | ||
| sparse-checkout-cone-mode: false | ||
| - uses: strands-agents/devtools/issue-labeler@5e50593798905abfbb31cc709f5ac58a2992b03a | ||
|
jesseturner21 marked this conversation as resolved.
|
||
| with: | ||
| aws_role_arn: ${{ secrets.BEDROCK_SECURITY_REVIEW_ROLE_ARN }} | ||
| config_path: '.github/labelers/severity.yml' | ||
| max_labels: '1' | ||
| # Severity calls for concrete evidence (stack traces, repro steps), | ||
| # which the default 1000-char truncation would often cut off. | ||
| max_body_length: '4000' | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.