Skip to content

fix(manifest): keep bid close independent of shutdown#410

Merged
Zblocker64 merged 1 commit into
mainfrom
support-435-manifest-timeout
Jul 23, 2026
Merged

fix(manifest): keep bid close independent of shutdown#410
Zblocker64 merged 1 commit into
mainfrom
support-435-manifest-timeout

Conversation

@chalabi2

@chalabi2 chalabi2 commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Summary

Testing

  • GOWORK=off go test ./manifest -count=1
  • GOWORK=off go test ./...

@chalabi2
chalabi2 requested a review from a team as a code owner June 22, 2026 15:22
@coderabbitai

coderabbitai Bot commented Jun 22, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 7094c5d1-5a90-41b7-b0b3-29555f7b2d25

📥 Commits

Reviewing files that changed from the base of the PR and between f6b3312 and 1c55090.

📒 Files selected for processing (8)
  • cmd/provider-services/cmd/flags.go
  • cmd/provider-services/cmd/run.go
  • config.go
  • manifest/config.go
  • manifest/service.go
  • manifest/watchdog.go
  • manifest/watchdog_test.go
  • service.go
🚧 Files skipped from review as they are similar to previous changes (8)
  • service.go
  • manifest/config.go
  • config.go
  • manifest/service.go
  • cmd/provider-services/cmd/flags.go
  • cmd/provider-services/cmd/run.go
  • manifest/watchdog.go
  • manifest/watchdog_test.go

Walkthrough

Introduces a configurable BroadcastTimeout for manifest watchdog close-bid broadcasts. Broadcasts now use an independent timeout context, with configuration wired through provider startup and CLI validation. Tests cover parent cancellation isolation and broadcast deadline handling.

Changes

Watchdog broadcast context isolation and configurable timeout

Layer / File(s) Summary
BroadcastTimeout config contract
manifest/config.go, config.go
Adds the default timeout constant and timeout fields to manifest and provider configuration.
Provider service timeout wiring
service.go, manifest/service.go
Propagates BroadcastTimeout into manifest service configuration and watchdog creation.
CLI timeout validation
cmd/provider-services/cmd/flags.go, cmd/provider-services/cmd/run.go
Uses the configured default for the flag and validates non-positive values before assignment.
Watchdog broadcast context isolation
manifest/watchdog.go
Uses an independent timeout context for close-bid broadcasts and waits for broadcast completion during shutdown.
Watchdog broadcast tests
manifest/watchdog_test.go
Adds controllable broadcast mocks and tests for parent cancellation and timeout behavior.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Watchdog
  participant BroadcastContext
  participant Session
  participant Lifecycle
  Watchdog->>BroadcastContext: Create independent timeout context
  Watchdog->>Session: BroadcastMsgs(close-bid)
  Lifecycle-->>Watchdog: Request shutdown
  Session-->>Watchdog: Return result or deadline error
  Watchdog->>Lifecycle: Signal shutdown completion
Loading

Poem

🐰 A timeout now guards the bid-close flight,
Parent shutdown cannot end it mid-night.
The watchdog waits, the broadcast runs free,
Then lifecycle closure follows peacefully.
Hop, hop—configurable timing agrees!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 18.18% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly matches the main change: keeping manifest bid close independent of watchdog shutdown.
Description check ✅ Passed The description is directly related to the changeset and the linked bug fix.
Linked Issues check ✅ Passed The PR addresses #435 by letting bid closing complete after manifest timeout instead of being canceled by watchdog shutdown.
Out of Scope Changes check ✅ Passed The changes stay focused on broadcast timeout handling, watchdog behavior, and supporting tests.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch support-435-manifest-timeout

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

troian
troian previously approved these changes Jul 21, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
cmd/provider-services/cmd/run.go (2)

832-834: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Validate the webhook port before narrowing it.

A configured value above 65535 can wrap at int32(webhookPort): registration may succeed with a different valid port while ListenAndServeTLS fails on the original invalid port. The fail-closed webhook then blocks tenant pod creation.

Proposed fix
  webhookPort := viper.GetInt(FlagAttestationWebhookPort)
+ if webhookPort < 1 || webhookPort > 65535 {
+   return fmt.Errorf("%w: %s must be between 1 and 65535",
+     errInvalidConfig, FlagAttestationWebhookPort)
+ }
  webhookAddr := fmt.Sprintf(":%d", webhookPort)

Also applies to: 916-919

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/provider-services/cmd/run.go` around lines 832 - 834, Validate the value
returned by viper.GetInt(FlagAttestationWebhookPort) in both webhook setup paths
before converting or passing it to any narrower integer type. Reject values
outside the valid TCP port range 1–65535 and fail closed before registration or
ListenAndServeTLS, keeping the validated value consistent across webhook
configuration.

Source: Linters/SAST tools


932-943: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Bound webhook deregistration during shutdown.

DeregisterWebhookConfiguration receives context.Background(), so an unavailable Kubernetes API can block shutdown indefinitely before webhookSrv.Shutdown runs. Use a timeout context for the delete as well.

Proposed fix
  <-ctx.Done()
+ deregisterCtx, deregisterCancel := context.WithTimeout(context.Background(), 5*time.Second)
+ defer deregisterCancel()
  if webhookKC, kcErr := fromctx.KubeClientFromCtx(ctx); kcErr == nil {
    attestwebhook.DeregisterWebhookConfiguration(
-     context.Background(), webhookKC, webhookLog,
+     deregisterCtx, webhookKC, webhookLog,
    )
  }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/provider-services/cmd/run.go` around lines 932 - 943, The shutdown
handler currently calls DeregisterWebhookConfiguration with an unbounded
context, which can delay webhook server shutdown indefinitely. Create and use a
bounded timeout context for deregistration in the group.Go shutdown flow,
ensuring the context is canceled appropriately, then preserve the existing
webhookSrv.Shutdown timeout behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@cmd/provider-services/cmd/run.go`:
- Around line 832-834: Validate the value returned by
viper.GetInt(FlagAttestationWebhookPort) in both webhook setup paths before
converting or passing it to any narrower integer type. Reject values outside the
valid TCP port range 1–65535 and fail closed before registration or
ListenAndServeTLS, keeping the validated value consistent across webhook
configuration.
- Around line 932-943: The shutdown handler currently calls
DeregisterWebhookConfiguration with an unbounded context, which can delay
webhook server shutdown indefinitely. Create and use a bounded timeout context
for deregistration in the group.Go shutdown flow, ensuring the context is
canceled appropriately, then preserve the existing webhookSrv.Shutdown timeout
behavior.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 3307ca19-41af-4c05-b8da-74998e76eb68

📥 Commits

Reviewing files that changed from the base of the PR and between b05aaae and f6b3312.

📒 Files selected for processing (8)
  • cmd/provider-services/cmd/flags.go
  • cmd/provider-services/cmd/run.go
  • config.go
  • manifest/config.go
  • manifest/service.go
  • manifest/watchdog.go
  • manifest/watchdog_test.go
  • service.go
🚧 Files skipped from review as they are similar to previous changes (5)
  • config.go
  • cmd/provider-services/cmd/flags.go
  • manifest/service.go
  • manifest/watchdog.go
  • manifest/watchdog_test.go

Manifest timeout close bids used a context tied to watchdog shutdown. If provider shutdown arrived after timeout fired, the broadcast could fail with context canceled and leave the bid open.

Use a fresh bounded tx context for the close-bid broadcast and wire the existing broadcast timeout into the manifest watchdog.

Refs akash-network/support#435

Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
@Zblocker64
Zblocker64 force-pushed the support-435-manifest-timeout branch from f6b3312 to 1c55090 Compare July 23, 2026 18:15
@Zblocker64
Zblocker64 merged commit 5a989d0 into main Jul 23, 2026
12 checks passed
@Zblocker64
Zblocker64 deleted the support-435-manifest-timeout branch July 23, 2026 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Bid is not closed after the manifest timeout

3 participants