Skip to content

Commit 68dacdb

Browse files
chrisleekrclaude
andauthored
fix(idempotency): gate side-effecting handlers with Valkey claim to prevent redelivery duplicates (#212)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 2c58ec1 commit 68dacdb

10 files changed

Lines changed: 232 additions & 8 deletions

File tree

CLAUDE.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ Single HTTP server (`src/app.ts`) using `octokit` App class. Webhook events arri
6262
## Key Concepts
6363

6464
- **Async processing**: Webhook must respond within 10 seconds. All heavy work runs asynchronously after 200 OK.
65-
- **Idempotency**: Two-layer guard. Fast path: in-memory `Map` keyed by `X-GitHub-Delivery` header (lost on restart). Durable: `isAlreadyProcessed()` checks GitHub for an existing tracking comment, survives pod restarts and OOM kills.
65+
- **Idempotency**: GitHub webhooks are at-least-once, a delivery (auto-retry or operator redelivery) replays with the SAME `X-GitHub-Delivery` header for up to 3 days. The four side-effecting event handlers (`events/issue-comment.ts`, `events/review-comment.ts`, and the label branches of `events/issues.ts` + `events/pull-request.ts`) call `claimDelivery(deliveryId)` (`src/webhook/idempotency.ts`) at the very top of their dispatch path, before any LLM call, `workflow_runs` insert, or GitHub write. `claimDelivery` is a Valkey `SET key 1 NX EX 259200` claim: it returns `true` exactly once per `deliveryId` within the 3-day window (the redelivery gets `false` and the handler returns early). It is **fail-OPEN**, when Valkey is unconfigured or errors it returns `true`, degrading to at-least-once rather than dropping webhooks. `events/review.ts` is intentionally NOT gated: it fires only an idempotent reactor wake (no dispatch/write). Durable backstop behind the best-effort Valkey layer: the `idx_workflow_runs_inflight` partial-unique index, which makes the dispatcher reject a second in-flight run for the same workflow+target even if the Valkey claim was skipped (fail-open). `claimDelivery` also fails open when Valkey is configured-but-disconnected (gated on `isValkeyHealthy()` so a down connection skips the SET rather than blocking on Bun's offline queue). (The legacy in-memory `Map` + `isAlreadyProcessed()` tracking-comment scan lives on the `router.ts` `processRequest` path, which the production handlers bypass, issue #202.)
6666
- **Repo checkout**: Each request clones the repo to a unique temp dir. Claude operates on local files via `cwd`.
6767
- **MCP servers**: Comment updates, inline reviews, and Context7 for library docs. Git changes are made via git CLI (Bash tool) on the cloned repo.
6868
- **Scheduled actions**: a repo may ship a `.github-app.yaml` at its default-branch root declaring prompt-based actions on a cron schedule. The internal scheduler (`src/scheduler/`, gated by `SCHEDULER_ENABLED` + `DATABASE_URL` + non-empty `ALLOWED_OWNERS`) enqueues a `scheduled-action` job, a new job kind on the scoped-job rail, that the daemon runs as one agent session via `src/daemon/scheduled-action-executor.ts`. Missed cron slots are skipped, not backfilled. The prompt is owner-trusted config. Cron parsing uses the `cron-parser` dependency. The bot-provided `merge_readiness` MCP tool is exposed only when `SCHEDULER_ALLOW_AUTO_MERGE` env AND per-action `auto_merge` are both true; `allowed_tools` is owner-trusted config, so an action granted a merge-capable Bash tool can still merge regardless. `resolve.ts` FR-017 is untouched.

docs/build/architecture.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ flowchart TD
99
GH["GitHub webhook<br/>POST /api/github/webhooks"]:::entry
1010
VERIFY["Verify HMAC-SHA256"]:::guard
1111
ACK["200 OK within 10 seconds"]:::ack
12-
ROUTE["Router<br/>idempotency + allowlist + concurrency"]:::guard
12+
ROUTE["Handler dispatch<br/>delivery claim + allowlist + concurrency"]:::guard
1313
TR["Haiku triage<br/>binary heavy classifier"]:::decide
1414
QUEUE["Orchestrator job queue<br/>Valkey list"]:::store
1515
SCALE{{"Scale-up decision<br/>heavy OR queue >= threshold<br/>AND no persistent slots<br/>AND cooldown elapsed"}}:::fork
@@ -44,7 +44,7 @@ flowchart TD
4444
## Key concepts
4545

4646
- **Async processing.** The webhook handler responds within ten seconds, so the router fires `processRequest` with fire-and-forget semantics after the 200 OK is queued. Every box downstream of `ACK` runs after the HTTP response is on the wire.
47-
- **Two-layer idempotency.** The fast path is an in-memory `Map` keyed by `X-GitHub-Delivery`. The durable path (`isAlreadyProcessed` in `src/core/tracking-comment.ts`) scans GitHub issue/PR comments for the hidden delivery marker the bot embeds in the tracking comment, so duplicate deliveries are detected across pod restarts, OOM kills, and crash loops: this works **without** `DATABASE_URL`. `DATABASE_URL` is required only to persist execution / dispatch history across restarts.
47+
- **Webhook delivery idempotency (issue #202).** GitHub is at-least-once: a delivery (auto-retry or operator redelivery) replays with the same `X-GitHub-Delivery` for up to 3 days. The four side-effecting handlers (`events/issue-comment.ts`, `events/review-comment.ts`, the label branches of `events/issues.ts` + `events/pull-request.ts`) call `claimDelivery(deliveryId)` (`src/webhook/idempotency.ts`) at the top of their dispatch path, before any LLM call, `workflow_runs` insert, or GitHub write. It is a Valkey `SET key 1 NX EX 259200` claim: `true` exactly once per delivery, `false` (and an early return) on a redelivery. It is **fail-open**, when Valkey is unconfigured or disconnected (gated on `isValkeyHealthy()`) it returns `true`, degrading to at-least-once rather than dropping or blocking webhooks. `events/review.ts` is exempt (idempotent reactor wake only). The durable backstop behind the best-effort Valkey layer is the `idx_workflow_runs_inflight` partial-unique index: the dispatcher rejects a second in-flight run for the same workflow+target even when the Valkey claim was skipped. The legacy in-memory `Map` + `isAlreadyProcessed` tracking-comment scan (`src/core/tracking-comment.ts`) now runs only on the `router.ts processRequest` path, which production handlers bypass. `DATABASE_URL` is required to persist execution / dispatch history and the in-flight guard across restarts.
4848
- **One request, one clone.** Each delivery clones the repo into a unique temp directory under `CLONE_BASE_DIR` **on the daemon host**. Claude operates on local files via `cwd`. On PR events the checkout supplementally fetches `origin/<baseBranch>` (when it differs from the head ref) so the agent's `git diff origin/<baseBranch>...HEAD` and `git rebase origin/<baseBranch>` directives resolve first try. A sibling `${workDir}-artifacts` directory is created outside the checkout and exposed to the agent as `BOT_ARTIFACT_DIR`: workflow summary files (IMPLEMENT.md / REVIEW.md / RESOLVE.md) are written there so they can never be picked up by a `git add` inside the clone. Both directories are removed in the pipeline's `finally` block regardless of outcome.
4949
- **GitHub credential resolution.** `src/core/github-token.ts:resolveGithubToken()` is the single source of the GitHub credential the daemon uses. Default is an App installation token minted just-in-time from the cached `App` singleton in `src/orchestrator/connection-handler.ts`. When `GITHUB_PERSONAL_ACCESS_TOKEN` is set, the helper short-circuits and returns the PAT instead, API/git authentication runs as the PAT owner. Commit author/committer metadata is **not** affected; `src/core/checkout.ts` hard-pins git `user.name`/`user.email` to `chrisleekr-bot[bot]` so commit objects still carry the bot identity. The git credential helper, executor `GH_TOKEN`/`GITHUB_TOKEN` env vars, and MCP server env all consume the resolved string without caring about its source.
5050
- **The webhook server never runs the pipeline.** Only daemons execute `runPipeline`. The webhook server is the orchestrator: it enqueues jobs and optionally spawns ephemeral daemons.

src/webhook/events/issue-comment.ts

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ import { addReaction } from "../../utils/reactions";
1010
import { dispatchByIntent } from "../../workflows/dispatcher";
1111
import { dispatchCommentSurface } from "../../workflows/ship/command-dispatch";
1212
import { isOwnerAllowed } from "../authorize";
13+
import { claimDelivery } from "../idempotency";
1314

1415
/**
1516
* Handler for issue_comment.created events.
@@ -84,6 +85,11 @@ export function handleIssueComment(
8485
// only on PR comments. Canonical wins; legacy `dispatchByIntent`
8586
// runs only when canonical produced no command.
8687
void (async (): Promise<void> => {
88+
// Idempotency gate (issue #202): GitHub redelivers with the same
89+
// deliveryId, so a redelivery would re-run both the canonical NL classifier
90+
// and the legacy intent LLM call (and any chat-thread turn). Claim the
91+
// delivery before any dispatch; a redelivery skips. Fail-open in claimDelivery.
92+
if (!(await claimDelivery(deliveryId, log))) return;
8793
const dispatchLog = log.child({ event_surface: eventSurface });
8894
let canonicalHandled = false;
8995
try {

src/webhook/events/issues.ts

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ import { dispatchByLabel } from "../../workflows/dispatcher";
77
import { dispatchCanonicalCommand } from "../../workflows/ship/command-dispatch";
88
import { routeTrigger } from "../../workflows/ship/trigger-router";
99
import { isOwnerAllowed } from "../authorize";
10+
import { claimDelivery } from "../idempotency";
1011

1112
// Permits hyphenated verbs (e.g. `bot:open-pr`, `bot:fix-thread`); the verb
1213
// must start with a letter and may contain `-`-separated lowercase segments.
@@ -97,6 +98,8 @@ export function handleIssues(octokit: Octokit, payload: IssuesEvent, deliveryId:
9798
const installationId = payload.installation?.id;
9899

99100
void (async (): Promise<void> => {
101+
// Idempotency gate (issue #202): skip a redelivery before any dispatch.
102+
if (!(await claimDelivery(deliveryId, log))) return;
100103
if (installationId !== undefined) {
101104
try {
102105
const command = await routeTrigger({

src/webhook/events/pull-request.ts

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ import { dispatchCanonicalCommand } from "../../workflows/ship/command-dispatch"
88
import { fireReactor } from "../../workflows/ship/reactor-bridge";
99
import { routeTrigger } from "../../workflows/ship/trigger-router";
1010
import { isOwnerAllowed } from "../authorize";
11+
import { claimDelivery } from "../idempotency";
1112

1213
// Permits the documented label shapes:
1314
// bot:ship, bot:abort-ship, bot:fix-thread, bot:investigate, ...
@@ -179,6 +180,8 @@ function handlePullRequestLabeled(
179180
const installationId = payload.installation?.id;
180181

181182
void (async (): Promise<void> => {
183+
// Idempotency gate (issue #202): skip a redelivery before any dispatch.
184+
if (!(await claimDelivery(deliveryId, log))) return;
182185
if (installationId !== undefined) {
183186
try {
184187
const command = await routeTrigger({

src/webhook/events/review-comment.ts

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ import { dispatchByIntent } from "../../workflows/dispatcher";
1111
import { dispatchCommentSurface } from "../../workflows/ship/command-dispatch";
1212
import { fireReactor } from "../../workflows/ship/reactor-bridge";
1313
import { isOwnerAllowed } from "../authorize";
14+
import { claimDelivery } from "../idempotency";
1415

1516
/**
1617
* Handler for pull_request_review_comment.{created,edited,deleted} events.
@@ -112,6 +113,8 @@ export function handleReviewComment(
112113
const threadId = String(topLevelCommentId);
113114

114115
void (async (): Promise<void> => {
116+
// Idempotency gate (issue #202): skip a redelivery before any LLM dispatch.
117+
if (!(await claimDelivery(deliveryId, log))) return;
115118
const dispatchLog = log.child({ thread_id: threadId, event_surface: "review-comment" });
116119
let canonicalHandled = false;
117120
try {

src/webhook/events/review.ts

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,12 @@ import { fireReactor } from "../../workflows/ship/reactor-bridge";
1010
*
1111
* Fires the ship reactor (T024) so any active intent on this PR wakes early
1212
* to inspect the new review state.
13+
*
14+
* No `claimDelivery` idempotency gate (issue #202): unlike the comment/label
15+
* handlers, this fires only an idempotent reactor wake (no LLM dispatch, no
16+
* workflow_runs row, no GitHub write). A redelivery just re-pokes an already-
17+
* awake intent, which is harmless and self-deduping, so the dedup claim would
18+
* add a Valkey round trip with nothing to protect.
1319
*/
1420
export function handleReview(
1521
_octokit: Octokit,

src/webhook/idempotency.ts

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
import type { Logger } from "pino";
2+
3+
import { getValkeyClient, isValkeyHealthy } from "../orchestrator/valkey";
4+
5+
/**
6+
* Webhook delivery idempotency (issue #202).
7+
*
8+
* GitHub replays a delivery (automatic retry or operator-driven manual
9+
* redelivery) with the SAME `X-GitHub-Delivery` header for up to 3 days, so
10+
* webhooks are at-least-once. Without a dedup gate, a redelivery re-runs the
11+
* full handler, including the triage/intent LLM calls and any chat-thread turn,
12+
* double-billing and posting duplicate replies. `claimDelivery` is the
13+
* canonical SET-NX-with-TTL idempotency claim, evaluated at the top of each
14+
* handler's dispatch path before any side-effect.
15+
*/
16+
17+
// 3 days, matching GitHub's redelivery window.
18+
const TTL_SECONDS = 259_200;
19+
const KEY_PREFIX = "idemp:webhook:";
20+
21+
/**
22+
* Claim a webhook delivery for processing.
23+
*
24+
* Returns `true` exactly once per `deliveryId` within the TTL window (the first
25+
* caller proceeds); a redelivery gets `false` and must skip.
26+
*
27+
* Fail-OPEN: if Valkey is unavailable or errors, returns `true` so an outage
28+
* degrades to at-least-once processing rather than dropping every webhook. The
29+
* `idx_workflow_runs_inflight` partial-unique index remains the durable backstop
30+
* against duplicate work when this best-effort layer is skipped: the dispatcher
31+
* rejects a second in-flight run for the same workflow+target. (The
32+
* tracking-comment marker scan via `isAlreadyProcessed` is NOT a backstop here:
33+
* it runs only on the legacy `router.ts processRequest` path that production
34+
* handlers bypass, issue #202.)
35+
*/
36+
export async function claimDelivery(deliveryId: string, log: Logger): Promise<boolean> {
37+
const client = getValkeyClient();
38+
// `getValkeyClient()` returns a non-null client even while the TCP connection
39+
// is down (it is null only when VALKEY_URL is unset). Bun's RedisClient
40+
// defaults to `enableOfflineQueue: true`, so issuing SET against a
41+
// disconnected client would QUEUE and block (up to the 10s connectionTimeout)
42+
// instead of failing open. Gate on `isValkeyHealthy()` (the same liveness
43+
// signal `router.ts` dispatch guards use) so a configured-but-down Valkey
44+
// takes the immediate fail-open path, leaving the durable backstops
45+
// (`idx_workflow_runs_inflight` + tracking-comment marker scan) to dedup.
46+
if (client === null || !isValkeyHealthy()) {
47+
log.warn({ deliveryId }, "claimDelivery: Valkey unavailable, proceeding (fail-open)");
48+
return true;
49+
}
50+
try {
51+
// SET key 1 NX EX <ttl>: returns "OK" iff the key did not exist (we won the
52+
// claim); returns null when it already exists (a redelivery).
53+
// `RedisClient.send` is typed `Promise<any>`; SET-NX returns "OK" or null.
54+
const res = (await client.send("SET", [
55+
`${KEY_PREFIX}${deliveryId}`,
56+
"1",
57+
"NX",
58+
"EX",
59+
String(TTL_SECONDS),
60+
])) as string | null;
61+
if (res === "OK") return true;
62+
log.info(
63+
{ deliveryId, event: "dedup-skip" },
64+
"claimDelivery: duplicate webhook delivery, skipping",
65+
);
66+
return false;
67+
} catch (err) {
68+
log.warn(
69+
{ deliveryId, err: err instanceof Error ? err.message : String(err) },
70+
"claimDelivery: Valkey error, proceeding (fail-open)",
71+
);
72+
return true;
73+
}
74+
}

test/webhook/events/issues.test.ts

Lines changed: 24 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,12 @@
1313
* to the dispatcher, which relies on the partial unique index at the
1414
* runs-store layer. The handler is invoked twice → dispatcher is invoked
1515
* twice → second call returns {status:"refused", reason:"in-flight…"}.
16+
*
17+
* Dispatch is fire-and-forget inside a `void (async () => {...})()` IIFE whose
18+
* first statement is `await claimDelivery(...)` (issue #202). That await defers
19+
* the `dispatchByLabel` call past the synchronous test body, so the positive
20+
* cases must drain the microtask queue (`flushMicrotasks`) before asserting,
21+
* otherwise the deferred call also leaks into the next test after `mockClear`.
1622
*/
1723

1824
import type { IssuesEvent } from "@octokit/webhooks-types";
@@ -47,6 +53,13 @@ const { handleIssues } = await import("../../../src/webhook/events/issues");
4753

4854
const fakeOctokit = {} as unknown as Octokit;
4955

56+
// Drain the microtask queue so the fire-and-forget dispatch IIFE runs past its
57+
// leading `await claimDelivery(...)` gate (#202) and the `await dispatchByLabel`
58+
// call lands before assertions / the next test's `mockClear`.
59+
async function flushMicrotasks(): Promise<void> {
60+
for (let i = 0; i < 5; i++) await Promise.resolve();
61+
}
62+
5063
function issueLabeledPayload(overrides?: {
5164
labelName?: string;
5265
senderLogin?: string;
@@ -76,8 +89,9 @@ describe("handleIssues", () => {
7689
);
7790
});
7891

79-
it("dispatches for bot:triage on open issue from allowed sender (T012)", () => {
92+
it("dispatches for bot:triage on open issue from allowed sender (T012)", async () => {
8093
handleIssues(fakeOctokit, issueLabeledPayload({ labelName: "bot:triage" }), "delivery-1");
94+
await flushMicrotasks();
8195

8296
expect(mockDispatchByLabel).toHaveBeenCalledTimes(1);
8397
const call = mockDispatchByLabel.mock.calls[0] as unknown as [
@@ -116,8 +130,13 @@ describe("handleIssues", () => {
116130
});
117131

118132
it("T014: duplicate label events delegate to dispatcher, second invocation is refused by idempotency guard", async () => {
119-
// First call: normal dispatch
133+
// First call: normal dispatch. Drain BEFORE arming the once-impl so this
134+
// dispatch resolves on the default "dispatched" path. Dispatch is deferred
135+
// past the leading `await claimDelivery(...)` gate (#202), so without this
136+
// drain the once-impl below would be consumed by THIS (first) call instead
137+
// of the second one.
120138
handleIssues(fakeOctokit, issueLabeledPayload({ labelName: "bot:triage" }), "delivery-dup-1");
139+
await flushMicrotasks();
121140

122141
// Second call with the same (target, workflow): dispatcher reports the
123142
// partial unique index rejection surfaced by runs-store.
@@ -130,9 +149,9 @@ describe("handleIssues", () => {
130149
);
131150
handleIssues(fakeOctokit, issueLabeledPayload({ labelName: "bot:triage" }), "delivery-dup-2");
132151

133-
// Let the micro-task queue drain so the fire-and-forget dispatch resolves.
134-
await Promise.resolve();
135-
await Promise.resolve();
152+
// Let the micro-task queue drain so the second fire-and-forget dispatch
153+
// resolves.
154+
await flushMicrotasks();
136155

137156
expect(mockDispatchByLabel).toHaveBeenCalledTimes(2);
138157
// Second call's resolved outcome is the "in-flight" refusal, the handler

0 commit comments

Comments
 (0)