Repository navigation
Release 1.0.0 - #16
Merged
Merged
Conversation
The ignore list keeps the maintainer's working notes, test output and local secrets out of the repository. The editor settings match the existing files: LF, UTF-8, 4 spaces for PowerShell, 2 for YAML and HTML, tabs for Go.
Keeps it for people who import the repository into Azure DevOps and want to run the engine tests there. Its path trigger and the analyzer settings comment now point at the new location.
aimetrics becomes github.com/antarsoft/ado-ai-code-review/metrics, so the packages split out of main.go later can be imported by path. The ignored build output name follows the new default binary name.
Phase 1: licence, root files, Azure DevOps CI moved to .azuredevops/, Go module renamed.
The public repository is antarsoft/adosentry. Updates the Go module path, the module manifest author and the example pipeline's repository reference. The ai-code-review/ folder and the AICodeReview module keep their names, because consuming pipelines reference those paths.
PowerShell job runs pinned Pester and PSScriptAnalyzer and uploads NUnit results; Go job runs gofmt, vet, test -race and build on metrics/; the secret scan runs the checksum-verified gitleaks binary over full history. Actions are pinned by SHA with a read-only token. Covers plan items C1-C5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vzP7qgCLcckiJRsmwb
Add GitHub Actions CI and Dependabot (Phase 2, C1–C5)
infra/main.bicep deploys a serverless Cosmos DB account (AIMetrics/Events, partition key /featureName) with disableLocalAuth, TLS 1.2 minimum and continuous backup, via modules/cosmos.bicep. bicepconfig.json raises the key linter rules to errors. The new CI job installs the standalone Bicep CLI pinned by version and SHA-256, then runs build and lint on every .bicep/.bicepparam under infra/. Covers plan items I1 and C7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vyww1vCteNVxjjjwRU
M10: the metrics image now copies go.mod and go.sum, runs go mod download and go mod verify (no go mod tidy), builds a trimmed static binary, and ships it on distroless static as the nonroot user. Both base images are pinned by digest, and a .dockerignore keeps the build context to the files the build needs. C6: a new docker job in CI builds the image without pushing, fails if it would run as root, and smoke-tests /healthz. Dependabot now also watches the Docker base images in metrics/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vzWJSUd2itrLB4GMxF
Add Bicep for Cosmos DB and a Bicep lint job in CI (I1, C7)
Harden metrics Dockerfile and build the image in CI (M10, C6)
Bumps the go-modules group with 2 updates in the /metrics directory: [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) and [github.com/Azure/azure-sdk-for-go/sdk/data/azcosmos](https://github.com/Azure/azure-sdk-for-go). Updates `github.com/Azure/azure-sdk-for-go/sdk/azcore` from 1.22.0 to 1.23.3 - [Release notes](https://github.com/Azure/azure-sdk-for-go/releases) - [Commits](Azure/azure-sdk-for-go@sdk/azcore/v1.22.0...sdk/azcore/v1.23.3) Updates `github.com/Azure/azure-sdk-for-go/sdk/data/azcosmos` from 1.4.2 to 1.5.0 - [Release notes](https://github.com/Azure/azure-sdk-for-go/releases) - [Commits](Azure/azure-sdk-for-go@sdk/data/azcosmos/v1.4.2...sdk/azcore/v1.5.0) --- updated-dependencies: - dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore dependency-version: 1.23.3 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-modules - dependency-name: github.com/Azure/azure-sdk-for-go/sdk/data/azcosmos dependency-version: 1.5.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-modules ... Signed-off-by: dependabot[bot] <support@github.com>
…-modules-815af11bde Bump the go-modules group across 1 directory with 2 updates
Covers what AdoSentry does, how the stage works, a 15-minute quick start (including the 'templates' alias requirement), rules files, cost notes, CI and the Apache-2.0 licence. Sections that later phases will change (parameter reference, troubleshooting, metrics auth and deployment, SECURITY.md, CONTRIBUTING.md) are marked with TODO placeholders. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4w1DgLw9LhDsXC5SZ42
The scripts hard-coded https://dev.azure.com/<org>/, so *.visualstudio.com organisations and Azure DevOps Server collections could not be reviewed. Get-ExistingReviewThreads, Set-PullRequestComments and Get-ReviewFeedbackReport now take -CollectionUri, with -Organization kept as a dev.azure.com fallback. The stage template passes $(System.CollectionUri). Tests cover the dev.azure.com, visualstudio.com and ADO Server URL shapes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vvgK6QLnp7mzVuuBbv
The bot can no longer vote on a pull request. Removes the AutoApprove switch, the reviewer-API PUT, the 'auto-approved' outcome and the AutoApproved result property, plus the connectionData call that existed only to find the reviewer id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vvgK6QLnp7mzVuuBbv
foundryAuth now defaults to entra, and FOUNDRY_KEY is mapped into the
step only when foundryAuth is key, so the key never reaches the step in
Entra mode.
An unset pipeline variable reaches the script as its own macro text.
Invoke-LLMCodeReview treats a Key starting with '$(' as empty (and fails
with a clear message in key mode), and the metrics call sends no
X-Api-Key header when aiMetricsApiKey is unset.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8A4vvgK6QLnp7mzVuuBbv
promptMode now defaults to rules, so promptFile is a short rules file and the engine adds its shared base prompt; legacy stays available. The stage now runs only when Build.Reason is PullRequest, so CI, manual and scheduled runs skip it instead of failing on missing System.PullRequest.* variables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vvgK6QLnp7mzVuuBbv
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4w1DgLw9LhDsXC5SZ42
Add initial README (X1)
String parameters placed inside '...' in the inline script (promptFile, reviewPathSpec, reviewExcludePathSpec, foundryTokenResource) now go through replace(..., '''', ''''''), so a single quote is doubled instead of ending the string. Branch, project, repository and PR id values were pasted into the script as macros, some unquoted. A source branch named with $(...) or a quote could run code in the review step, and a project name with a space broke the call. They now reach the script through AICR_* environment variables. Tests expand the template with quote-containing values and parse the result as PowerShell. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vvgK6QLnp7mzVuuBbv
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vvgK6QLnp7mzVuuBbv
Phase 3 engine fixes: Entra default, any ADO host, no AutoApprove, safe inline script (E1–E9)
The quick start no longer names a specific model. It says any chat model with JSON-schema structured output works, that modelName is the deployment name, and that the dashboard shows $0 cost for a name that is not in the metrics pricing table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4w1DgLw9LhDsXC5SZ42
README: use a placeholder model name instead of suggesting one
Pin the HTTP contract before the package split: status codes, bodies and headers for every route, the stored event envelope, the Cosmos query and its parameters, and the full /metrics/summary response as a golden file with dates written relative to today. The tests drive the real router and the real azcosmos client against an in-memory Cosmos fake (internal/cosmostest), so they need no Azure account and keep passing unchanged while the code moves into packages. main.go only gains newContainerClient and newHandler, so tests and main share the same wiring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vwf6graWpMZ1X3qQkX
… table tests (M2, M3, M4) main.go now only loads configuration and wires the pieces together: - internal/config: environment variables, every missing one reported at once - internal/storage: the Store interface and its Cosmos implementation - internal/summary: pure aggregation (Aggregator.Add / Result) with the clock and price table passed in - internal/pricing: model rates in an embedded pricing.json; PRICING_FILE replaces them, and a malformed file fails at startup - internal/handlers: the HTTP routes, one per line so middleware can wrap a single route (Phase 5 auth) - internal/web: the embedded pages The M1 characterisation tests pass unchanged, including the golden summary, so the HTTP contract and stored event shape are the same. New table-driven tests cover aggregation, cost (unknown model = $0), day ranges, percentile, the lenient field readers, pricing files and configuration. The unused boolFromAny helper is dropped. The Dockerfile copies internal/ and the .dockerignore keeps tests and the Cosmos fake out of the image. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vwf6graWpMZ1X3qQkX
totalTokens.cached in GET /metrics/summary was always 0: the summary never added up custom.cachedTokens, and the engine never sent it. The engine now reads usage.prompt_tokens_details.cached_tokens per batch, the stage sends the total as cachedTokens, and the summary sums it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vwUafsCpakSBTXzjqU
… M7, M13, M15) New internal/auth package wraps POST /events only: - entra: validates an Entra ID access token's RS256 signature against the tenant's signing keys (cached a day, refreshed early on an unknown kid at most every 5 minutes, kept through an Entra outage), then iss (v1 or v2), aud, exp/nbf with 2 minutes leeway and tid. The token must carry the Events.Write app role, or come from an oid/azp on ENTRA_ALLOWED_PRINCIPALS; otherwise 403. - key: X-Api-Key compared in constant time against METRICS_API_KEY (at least 32 characters). - AUTH_MODE picks entra, key, entra,key or none. It has no default: the service will not start without it, and none logs a warning. A handler wired without auth refuses writes with 503. Table-driven tests cover valid, expired, not-yet-valid, wrong audience, issuer and tenant, missing role, allowlist, forged signature, HS256 and alg none, key rollover, bad and missing keys, both modes and none. The Docker smoke test now starts in entra mode and checks an unauthenticated write gets 401. send-event-curl.sh and the main.go header document the credentials. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vwUafsCpakSBTXzjqU
New metricsAuth template parameter, default entra. In entra mode the stage gets an access token for aiMetricsApiResource (the API's application ID URI) with the service connection's identity and sends it as a bearer token; the token is registered as a pipeline secret. In key mode it sends aiMetricsApiKey as X-Api-Key, and only that mode maps the key into the step. Get-MetricsRequestHeaders (exported) builds the header and refuses a missing setting, which the stage logs as a metrics warning. README documents AUTH_MODE and the metrics credentials; CHANGELOG notes the auth change and the totalTokens.cached fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vwUafsCpakSBTXzjqU
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vwUafsCpakSBTXzjqU
Metrics auth: Entra token or API key on POST /events (Phase 5) + cached token fix
Sentry.io holds registered marks for developer tooling, and the VS Marketplace asks for "Azure DevOps" in full, so the project is renamed before 1.0. Listings will read "PullPicket for Azure DevOps". - Go module path is now github.com/antarsoft/pullpicket/metrics - Bicep namePrefix and application tag default to pullpicket - README, CI, examples, engine help text, tests and logo label updated - CHANGELOG entry under Unreleased Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vvkuA2ionQD35Yc64U
Rename AdoSentry to PullPicket
Basic ACR with no admin user and no anonymous pull. A user-assigned identity holds AcrPull on it, created before the Container App so the app's first image pull already has the grant (a system-assigned identity would only exist after that pull). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vwVykbRMCMnpzV6KzL
Adds a Consumption Container Apps environment with Log Analytics, and the Container App itself: external HTTPS ingress on 8080, /healthz probes, scale to zero by default, image pulled from ACR with the user-assigned identity. The identity gets the Cosmos DB Built-in Data Contributor data-plane role through sqlRoleAssignments (key auth stays disabled), and the app finds it through AZURE_CLIENT_ID. The app gets AUTH_MODE (default entra), ENTRA_TENANT_ID (this tenant by default) and ENTRA_AUDIENCE, without which it refuses to start; key mode stores METRICS_API_KEY as a Container Apps secret. The app is only deployed when imageTag is set, so the first deploy creates the registry and environment, and reports the app's future URL for the Entra redirect URI. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vwVykbRMCMnpzV6KzL
Bicep cannot create app registrations (D8), so an az CLI script does: - <prefix>-metrics-api with the Events.Write application role (service principals only) and an application ID URI, api://<prefix>-metrics by default, falling back to api://<client id> where the tenant requires it. - <prefix>-metrics-dashboard, the Container Apps sign-in app, with its redirect URI from the first deployment's metricsUrl, assignment required, and the signed-in user (or --dashboard-group) assigned. It is safe to rerun: registrations are found by name, and the dashboard client secret is only added on creation or with --rotate-secret. Results go to .env.entra (mode 600, git-ignored) as export lines for main.bicepparam. The Bicep CI job now also runs shellcheck over infra/*.sh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vwVykbRMCMnpzV6KzL
The Container App gets an authConfigs/current with the Entra provider (the dashboard app registration from setup-entra.sh, issuer pinned to the tenant): unauthenticated requests are sent to sign-in, except excludedPaths /events (which checks its own credential, AUTH_MODE) and /healthz. Who may sign in is set in Entra: the dashboard app requires assignment, so only assigned users or groups get through. The client secret is a Container Apps secret. main.bicepparam reads the image tag, Entra IDs and that secret from the environment (the .env.entra file setup-entra.sh writes), so none of them land in a file in the repo. The app module requires both dashboard values, so the app cannot be deployed without sign-in. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vwVykbRMCMnpzV6KzL
Assigns the API's Events.Write app role to the service principal behind an Azure DevOps service connection (or any sender), given its client id or object id. It finds the API from .env.entra or by name, checks the role exists, and skips the grant if it is already there. The header documents the equivalent single az rest call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vwVykbRMCMnpzV6KzL
infra/README.md walks through the two-pass deploy: base resources and the app's URL, Entra registration, image build with az acr build, the app, the pipeline's Events.Write grant and its variable group, and a check that sign-in and the 401 on POST /events work. It also lists outputs, options and teardown, including the app registrations. The root README now links it instead of the in-progress note, and the CHANGELOG records the new infrastructure and scripts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vwVykbRMCMnpzV6KzL
Phase 6: deploy the metrics infrastructure (I2–I7)
tools/Update-ParameterReference.ps1 reads the parameters: block of ai-code-review/stage-template.yml (name, type, default, allowed values, and the comment lines above each parameter as its description) and writes docs/parameters.md. With -Check it writes nothing and fails when the file is stale; the PowerShell CI job runs it that way, and PSScriptAnalyzer now covers tools/ too. The template gains comments for the parameters that had none, so every row has a description. The README links the reference instead of the placeholder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vweKZ8kvVx35iDBhzj
docs/troubleshooting.md covers the first-setup problems: the stage skipped on non-PR runs, comments refused because the service connection's identity is not in the Azure DevOps organisation or cannot contribute to pull requests (the step can still end green with no-comments), 401/403 from the model without the OpenAI User role, foundryTokenResource for classic Azure OpenAI, missing or overridden variables and self-hosted agent pools, plus a table of the engine's other error messages. The README links it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vweKZ8kvVx35iDBhzj
One page for the optional metrics service: the deploy steps in short (linking infra/README.md), connecting the pipeline's variable group, the four AUTH_MODE values with how the Entra check and the API key work and how to test a credential, what each dashboard panel shows, tenants, the pricing file and how to replace it, every environment variable, running locally, and a troubleshooting table. The README's metrics section shrinks to a summary that links it, so the environment variable table lives in one place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vweKZ8kvVx35iDBhzj
The security policy covers supported versions; private reporting through GitHub's "Report a vulnerability" with response aims; exactly what each model request contains (system prompt and rules, prior review rounds with the first 150 characters of the bot's comments, branch names, file paths, numbered diff with 5 lines of context, deleted file paths when enabled) and what it does not; the prompt-injection model and its limits, including marker-tagged threads reaching the history block; every field of the metrics event, including branch names and commented file paths; credential handling; and a hardening checklist. The README's security section now summarises it and links it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vweKZ8kvVx35iDBhzj
CONTRIBUTING.md covers the layout, local setup with the CI tool versions, the checks to run per area (including regenerating the parameter reference), pull request expectations, commit style as used in this repository, and the Developer Certificate of Origin. A new "DCO sign-off" CI job runs on pull requests and fails any non-merge commit without a Signed-off-by line for its author's email. Dependabot's and the AI assistant's commits are exempt, since a maintainer reviews and merges them. The README lists the job, links CONTRIBUTING.md and adds docs/ and tools/ to the layout; the changelog records the new documentation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vweKZ8kvVx35iDBhzj
docs/assets/pr-review.png shows two PullPicket comment threads on a pull request diff, a WARNING with a suggested fix and a SUGGESTION, in the format the engine posts (severity emoji and word, then the text). It is a rendered illustration of a placeholder repository (contoso-web) and placeholder code, not a capture of a real project. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4vweKZ8kvVx35iDBhzj
Phase 7: documentation (X2–X7)
Moves the unreleased notes under [1.0.0] - 2026-11-01 and adds a "Breaking defaults at a glance" table: the rename to PullPicket, foundryAuth=entra, promptMode=rules, AutoApprove removed, AUTH_MODE required, metricsAuth=entra, -CollectionUri, and the metrics store and API changes (partition key, CORS, totalTokens.cached). Adds compare and release links. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4w121dBr3nYoutNLaV7 Signed-off-by: Claude <noreply@anthropic.com>
The engine version restarts at 1.0.0 for the first PullPicket release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4w121dBr3nYoutNLaV7 Signed-off-by: Claude <noreply@anthropic.com>
Release 1.0.0: CHANGELOG entry and ModuleVersion (R1, R2)
The engine version in metrics events is the templates repository's git tag at HEAD, or its short commit, not ModuleVersion. The end-to-end test (R3) showed the commit hash, so the 1.0.0 intro now says that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4w121dBr3nYoutNLaV7 Signed-off-by: Claude <noreply@anthropic.com>
- Quick start: a new project's default repository is named after the project; classic Azure OpenAI needs foundryTokenResource; adding a service principal needs the org connected to Entra; the automatic connection's Contributor role is not needed. - infra/README: register the resource providers first; setup-entra.sh may fall back to api://<client id>. - Troubleshooting: "Given email address is invalid" when adding the identity, and "repository could not be retrieved". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8A4w121dBr3nYoutNLaV7 Signed-off-by: Claude <noreply@anthropic.com>
Fix docs found in the 1.0.0 end-to-end test (R3)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Ziad · project thread
Brings
mainup todevelopmentat240f9fc(merge of #15) for the first PullPicket release.mainis an ancestor ofdevelopment, so there are no conflicts.Merge with "Create a merge commit" (not squash or rebase), so the
v1.0.0tag points at the history CI tested.Release checklist (Phase 8)
[1.0.0] - 2026-11-01with the breaking defaults (Release 1.0.0: CHANGELOG entry and ModuleVersion (R1, R2) #14, corrected in Fix docs found in the 1.0.0 end-to-end test (R3) #15)ModuleVersion = '1.0.0'(Release 1.0.0: CHANGELOG entry and ModuleVersion (R1, R2) #14)git checkout main && git pull && git tag -a v1.0.0 -m "v1.0.0" && git push origin v1.0.0(Ziad)See CHANGELOG.md for the full 1.0.0 notes. Dependabot #2 and #5 are deliberately left for after the tag.
Generated by Claude Code