Skip to content

fix(schedule): bound preload startup wait - #213

Merged
smiggleworth merged 5 commits into
mainfrom
fix/schedule-preload-timeout
Aug 24, 2026
Merged

fix(schedule): bound preload startup wait#213
smiggleworth merged 5 commits into
mainfrom
fix/schedule-preload-timeout

Conversation

@smiggleworth

@smiggleworth smiggleworth commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Summary

  • replace the one-second Schedule preload reply deadline with a bounded 120-second startup watchdog
  • allow deployments to configure the watchdog through FITZ_SCHEDULE_PRELOAD_TIMEOUT_SECS
  • log preload start, persisted-family discovery, per-family debug progress, completion time, and timeout
  • preserve fail-closed behavior when the Schedule actor exits or the watchdog expires
  • add regressions for both a reply delayed beyond one second and an actor that misses an injected deadline
  • update recovery/configuration docs and satisfy the Rust 1.98 Clippy diagnostics from the prior CI run
  • cancel superseded backend CI runs using workflow- and PR/branch-scoped concurrency

Why

Cloud recovery and the Midge primary lease can complete successfully while Schedule preload is still queued behind actor work. Fitz previously treated a one-second actor reply delay as a preload failure and exited, causing an ECS restart loop even though the exclusive-writer lease had already been acquired.

An unbounded reply wait would remove that false failure but introduce a worse liveness regression when storage or the actor remains alive but stalls. This change gives S3-backed recovery a realistic default budget while retaining a finite, observable startup failure.

Schedule is the only domain with this exact fatal startup reply wait. Other one-second domain waits remain bounded runtime delivery, cleanup, admin, or observability operations. Queue and Stream perform boot storage preparation directly and do not add this aggregate actor-reply deadline.

Midge still has unbounded internal runtime response waits that Fitz cannot interrupt once entered. That follow-up is tracked in cntryl/midge#254.

Validation

  • cargo fmt --all -- --check
  • cargo clippy --locked --workspace --all-targets --all-features -- -D warnings -D clippy::pedantic
  • cargo test --locked --workspace
  • cargo test --locked --lib domains::schedule::sink::tests (25 passed)
  • actionlint .github/workflows/ci-backend.yml
  • focused red/green regression: actor blocked past an injected 20ms deadline fails explicitly
  • focused slow-success regression: actor blocked longer than one second still completes before the configured watchdog

All local validation is green.

@smiggleworth smiggleworth changed the title fix(schedule): wait for preload completion fix(schedule): bound preload startup wait Aug 24, 2026
@smiggleworth
smiggleworth marked this pull request as ready for review August 24, 2026 18:40
Copilot AI lite review requested due to automatic review settings August 24, 2026 18:40

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@smiggleworth
smiggleworth merged commit dc371e5 into main Aug 24, 2026
1 check failed
@smiggleworth
smiggleworth deleted the fix/schedule-preload-timeout branch August 24, 2026 18:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants