diff --git a/docs/development/architecture.md b/docs/development/architecture.md index d245e679..b66d14dc 100644 --- a/docs/development/architecture.md +++ b/docs/development/architecture.md @@ -829,10 +829,10 @@ seconds as a baseline, not an architectural upper bound. After activation, the shutdown coordinator also monitors Midge writer-lease renewal health. When Midge reports the lease unhealthy, Fitz withdraws orchestration health and strict readiness and requests fatal termination without -attempting in-process lease reacquisition. The pinned Midge revision still needs -an independent monotonic pre-TTL watchdog for blocked cloud renewal and -fail-closed parsing for malformed cloud lease expiration; Fitz cannot recreate -those lease-internal guarantees from the exposed boolean. +attempting in-process lease reacquisition. Midge independently enforces the +monotonic lease-valid-until deadline during provider renewal and fails closed on +malformed cloud lease expiration. Fitz configures that deadline through +`FITZ_STORAGE_LEASE_TTL_SECS`; Midge renews at one third of the TTL. ### Configuration **Type:** `BootConfig` ```rust diff --git a/docs/operations/cloud-setup.md b/docs/operations/cloud-setup.md index 6edfb575..f9125c1b 100644 --- a/docs/operations/cloud-setup.md +++ b/docs/operations/cloud-setup.md @@ -75,6 +75,15 @@ Expect Fitz to reach readiness after storage startup and writer-lease acquisitio Any other value is rejected at startup. +## Storage Writer Lease + +`FITZ_STORAGE_LEASE_TTL_SECS` configures the embedded Midge primary +storage-writer lease and defaults to `30`. Values below 30 seconds are rejected. +Midge renews the provider-backed lease at one third of the configured TTL, so +increasing the TTL reduces coordination requests while extending takeover time +after a crash or failed release. A graceful shutdown conditionally expires the +lease immediately; it does not wait for the TTL. + Schedule uses this policy for server-selected durable writes. KV and Stream still honor client-selected buffered versus sync intent, translated to cloud-compatible commits: buffered intent uses asynchronous cloud durability, diff --git a/docs/operations/observability.md b/docs/operations/observability.md index c47e3470..99c0442e 100644 --- a/docs/operations/observability.md +++ b/docs/operations/observability.md @@ -61,12 +61,13 @@ invariant rather than deriving behavior from logs. When Midge reports the lease unhealthy, Fitz withdraws `/targetz`, `/readyz`, and `/healthz` and requests fatal broker termination; the process does not attempt in-process reacquisition. -The pinned Midge revision does not yet expose or independently enforce a -monotonic lease-valid-until deadline while a provider renewal call is blocked; -its cloud calls can outlast the 30-second lease TTL. It also treats a malformed -cloud lease expiration as expired. Until Midge supplies a deadline watchdog and -fail-closed expiration parsing, do not treat provider-backed cloud takeover as -split-brain-safe under a stalled provider or corrupt lease object. +Midge independently enforces a monotonic lease-valid-until deadline while a +provider renewal is in flight and treats malformed cloud lease expiration as +indeterminate rather than expired. `FITZ_STORAGE_LEASE_TTL_SECS` configures the +deadline, defaults to 30 seconds, and cannot be set below 30 seconds. Midge +renews at one third of that TTL. Longer values reduce provider coordination +traffic but extend ungraceful takeover; graceful shutdown conditionally expires +the current lease immediately. --- diff --git a/docs/operations/operations-runbook.md b/docs/operations/operations-runbook.md index 6f7acd52..eea32fb6 100644 --- a/docs/operations/operations-runbook.md +++ b/docs/operations/operations-runbook.md @@ -79,11 +79,13 @@ withdraws `/targetz`, `/healthz`, and `/readyz` and terminates through the fatal path. Do not wait for a second signal and do not expect that process to reacquire the lease; verify that the orchestrator starts or promotes a healthy replacement. -Provider-backed cloud failover has an upstream safety gate in the pinned Midge -revision: renewal health has no independent monotonic deadline while provider -I/O blocks, and malformed cloud lease expiration is treated as expired. Do not -approve cloud takeover as split-brain-safe until Midge adds a pre-TTL watchdog -and fail-closed expiration parsing. +Provider-backed cloud renewal has an independent monotonic deadline in Midge, +and malformed lease expiration fails closed as indeterminate. The storage +writer lease TTL defaults to 30 seconds and is configured with +`FITZ_STORAGE_LEASE_TTL_SECS`; Midge renews at one third of that value. A longer +TTL reduces provider coordination requests but delays takeover after a crash or +failed release. Successful graceful shutdown conditionally expires the lease +immediately. ## Emergency Rollback diff --git a/docs/user-guides/vars.md b/docs/user-guides/vars.md index 8ec636ac..4173fedd 100644 --- a/docs/user-guides/vars.md +++ b/docs/user-guides/vars.md @@ -130,6 +130,7 @@ For the auth and browser-perimeter checklist, see | FITZ_STORAGE_PREFIX | Prefix string | Unset | Optional object key namespace prefix in cloud mode. | | FITZ_STORAGE_CACHE_PATH | Filesystem path | ./.fitz-cloud-cache | Local cache path for cloud-backed storage mode. | | FITZ_STORAGE_CLOUD_DURABILITY | background or strict | background | Cloud policy for broker-selected durable writes and client sync intent. `background` completes at Midge's local cloud commit barrier and uploads asynchronously; `strict` waits for provider acknowledgement. | +| FITZ_STORAGE_LEASE_TTL_SECS | Integer seconds, at least 30 | 30 | TTL for the embedded Midge primary storage-writer lease. Midge renews at one third of this interval. Increasing it reduces provider lease traffic but extends takeover after an ungraceful failure; graceful shutdown explicitly releases the lease. | | FITZ_STORAGE_MEMTABLE_BYTES | Unsigned integer byte count | Auto | Optional explicit memtable size override for embedded engine. | | FITZ_SCHEDULE_PRELOAD_TIMEOUT_SECS | Positive integer second count | 120 | Maximum aggregate wait for required Schedule actor preload during startup. Expiry fails startup with an explicit timeout rather than leaving the broker wedged indefinitely. | | FITZ_QUEUE_WRITE_POLICY | fast, buffered, or strict | fast | Queue mutation write policy. `fast` skips WAL and flushes in the background; `buffered` uses local buffered WAL or cloud asynchronous durability; `strict` waits for local sync or cloud provider acknowledgement. | diff --git a/src/boot/runtime/config.rs b/src/boot/runtime/config.rs index d0153f5a..2b651305 100644 --- a/src/boot/runtime/config.rs +++ b/src/boot/runtime/config.rs @@ -12,6 +12,7 @@ const DEFAULT_SQRZL_EMULATOR_BUCKET: &str = "fitz"; const DEFAULT_METRICS_BIND_ADDR: &str = "127.0.0.1"; const DEFAULT_METRICS_PORT: u16 = 9090; const ENV_STORAGE_MEMTABLE_BYTES: &str = "FITZ_STORAGE_MEMTABLE_BYTES"; +const ENV_STORAGE_LEASE_TTL_SECS: &str = "FITZ_STORAGE_LEASE_TTL_SECS"; const ENV_QUEUE_WRITE_POLICY: &str = "FITZ_QUEUE_WRITE_POLICY"; const ENV_QUEUE_LOSS_WINDOW_MS: &str = "FITZ_QUEUE_LOSS_WINDOW_MS"; const ENV_KV_IDLE_TRANSACTION_TTL_SECS: &str = "FITZ_KV_IDLE_TRANSACTION_TTL_SECS"; @@ -19,6 +20,8 @@ const ENV_SCHEDULE_PRELOAD_TIMEOUT_SECS: &str = "FITZ_SCHEDULE_PRELOAD_TIMEOUT_S const ENV_DRAIN_GRACE_SECONDS: &str = "FITZ_DRAIN_GRACE_SECONDS"; const ENV_DRAIN_CLOSE_REASON: &str = "FITZ_DRAIN_CLOSE_REASON"; const DEFAULT_QUEUE_LOSS_WINDOW_MS: u64 = 100; +const DEFAULT_STORAGE_LEASE_TTL_SECS: u64 = 30; +const MIN_STORAGE_LEASE_TTL_SECS: u64 = 30; const DEFAULT_KV_IDLE_TRANSACTION_TTL_SECS: u64 = 300; const DEFAULT_DRAIN_GRACE_SECONDS: u64 = 25; const DEFAULT_DRAIN_CLOSE_REASON: &str = "broker draining for redeploy"; @@ -328,7 +331,7 @@ mod env; use env::{ drain_close_reason_from_env, drain_grace_seconds_from_env, env_non_empty, kv_idle_transaction_ttl_seconds_from_env, queue_loss_window_ms_from_env, required_env, - schedule_preload_timeout_seconds_from_env, + schedule_preload_timeout_seconds_from_env, storage_lease_ttl_seconds_from_env, }; #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -444,6 +447,15 @@ impl<'a> StorageConfig<'a> { format!("{ENV_SCHEDULE_PRELOAD_TIMEOUT_SECS} must be greater than 0").into(), ); } + if let Some(error) = &config.storage_lease_ttl_error { + return Err(error.clone().into()); + } + if config.storage_lease_ttl_seconds < MIN_STORAGE_LEASE_TTL_SECS { + return Err(format!( + "{ENV_STORAGE_LEASE_TTL_SECS} must be at least {MIN_STORAGE_LEASE_TTL_SECS} seconds" + ) + .into()); + } config .storage_memtable .validate() @@ -519,6 +531,9 @@ pub struct BootConfig { pub cloud_durability: CloudDurabilityMode, /// Optional explicit Midge memtable size in bytes. pub storage_memtable: StorageMemtableConfig, + /// TTL for the embedded Midge primary storage-writer lease. + pub storage_lease_ttl_seconds: u64, + pub(crate) storage_lease_ttl_error: Option, /// Commit policy for queue durable mutations. pub queue_write_policy: QueueWritePolicy, /// Whether queue write policy was explicit or resolved through the default. @@ -676,6 +691,8 @@ impl Default for BootConfig { kv_idle_transaction_ttl_seconds_from_env(); let (schedule_preload_timeout_seconds, schedule_preload_timeout_error) = schedule_preload_timeout_seconds_from_env(); + let (storage_lease_ttl_seconds, storage_lease_ttl_error) = + storage_lease_ttl_seconds_from_env(); let (queue_write_policy, queue_write_policy_source) = QueueWritePolicy::from_env_with_source(); let drain_close_reason = drain_close_reason_from_env(); @@ -704,6 +721,8 @@ impl Default for BootConfig { channel_capacity: 1000, cloud_durability: CloudDurabilityMode::from_env(), storage_memtable: StorageMemtableConfig::from_env(), + storage_lease_ttl_seconds, + storage_lease_ttl_error, queue_write_policy, queue_write_policy_source, queue_loss_window_ms, @@ -889,6 +908,20 @@ impl BootConfig { self.storage_memtable.bytes() } + #[must_use] + /// Override the embedded Midge primary storage-writer lease TTL. + pub fn with_storage_lease_ttl_seconds(mut self, seconds: u64) -> Self { + self.storage_lease_ttl_seconds = seconds; + self.storage_lease_ttl_error = None; + self + } + + #[must_use] + /// Return the configured embedded Midge primary storage-writer lease TTL. + pub fn storage_lease_ttl(&self) -> Duration { + Duration::from_secs(self.storage_lease_ttl_seconds) + } + /// Validate the full broker boot configuration. /// /// # Errors diff --git a/src/boot/runtime/config/env.rs b/src/boot/runtime/config/env.rs index 8e617e47..eba64d20 100644 --- a/src/boot/runtime/config/env.rs +++ b/src/boot/runtime/config/env.rs @@ -1,7 +1,8 @@ use super::{ DEFAULT_DRAIN_CLOSE_REASON, DEFAULT_DRAIN_GRACE_SECONDS, DEFAULT_KV_IDLE_TRANSACTION_TTL_SECS, - DEFAULT_QUEUE_LOSS_WINDOW_MS, ENV_DRAIN_CLOSE_REASON, ENV_DRAIN_GRACE_SECONDS, - ENV_KV_IDLE_TRANSACTION_TTL_SECS, ENV_QUEUE_LOSS_WINDOW_MS, ENV_SCHEDULE_PRELOAD_TIMEOUT_SECS, + DEFAULT_QUEUE_LOSS_WINDOW_MS, DEFAULT_STORAGE_LEASE_TTL_SECS, ENV_DRAIN_CLOSE_REASON, + ENV_DRAIN_GRACE_SECONDS, ENV_KV_IDLE_TRANSACTION_TTL_SECS, ENV_QUEUE_LOSS_WINDOW_MS, + ENV_SCHEDULE_PRELOAD_TIMEOUT_SECS, ENV_STORAGE_LEASE_TTL_SECS, }; pub(super) fn env_non_empty(key: &str) -> Option { @@ -92,6 +93,14 @@ pub(super) fn schedule_preload_timeout_seconds_from_env() -> (u64, Option (u64, Option) { + positive_u64_from_env( + ENV_STORAGE_LEASE_TTL_SECS, + DEFAULT_STORAGE_LEASE_TTL_SECS, + "second count", + ) +} + pub(super) fn drain_close_reason_from_env() -> String { env_non_empty(ENV_DRAIN_CLOSE_REASON).unwrap_or_else(|| DEFAULT_DRAIN_CLOSE_REASON.to_string()) } diff --git a/src/boot/runtime/config/tests/base_auth_and_network.rs b/src/boot/runtime/config/tests/base_auth_and_network.rs index 3cc37bf2..2dd5274e 100644 --- a/src/boot/runtime/config/tests/base_auth_and_network.rs +++ b/src/boot/runtime/config/tests/base_auth_and_network.rs @@ -94,6 +94,7 @@ pub(super) fn with_storage_env(values: &[(&str, &str)], test: impl FnOnce() - "FITZ_STORAGE_NAMESPACE", "FITZ_STORAGE_ACCOUNT", "FITZ_STORAGE_CLOUD_DURABILITY", + "FITZ_STORAGE_LEASE_TTL_SECS", "FITZ_STORAGE_MEMTABLE_BYTES", ENV_QUEUE_WRITE_POLICY, ENV_QUEUE_LOSS_WINDOW_MS, diff --git a/src/boot/runtime/config/tests/cloud_storage.rs b/src/boot/runtime/config/tests/cloud_storage.rs index 7d36ac07..7bdac532 100644 --- a/src/boot/runtime/config/tests/cloud_storage.rs +++ b/src/boot/runtime/config/tests/cloud_storage.rs @@ -1,5 +1,70 @@ use super::*; +#[test] +#[serial] +fn should_default_storage_lease_ttl_to_thirty_seconds() { + with_storage_env(&[], || { + // Arrange + + // Act + let config = BootConfig::default(); + + // Assert + assert_eq!(config.storage_lease_ttl(), Duration::from_secs(30)); + }); +} + +#[test] +#[serial] +fn should_read_storage_lease_ttl_from_environment() { + with_storage_env(&[("FITZ_STORAGE_LEASE_TTL_SECS", "59")], || { + // Arrange + + // Act + let config = BootConfig::default().with_auth_config(crate::auth::AuthConfig::Disabled); + + // Assert + assert_eq!(config.storage_lease_ttl(), Duration::from_secs(59)); + assert!(config.validate().is_ok()); + }); +} + +#[test] +#[serial] +fn should_reject_storage_lease_ttl_below_current_safety_floor() { + with_storage_env(&[("FITZ_STORAGE_LEASE_TTL_SECS", "29")], || { + // Arrange + + // Act + let result = BootConfig::default().validate(); + + // Assert + assert!(result.is_err()); + assert!(result + .unwrap_err() + .to_string() + .contains("FITZ_STORAGE_LEASE_TTL_SECS must be at least 30 seconds")); + }); +} + +#[test] +#[serial] +fn should_reject_non_numeric_storage_lease_ttl() { + with_storage_env(&[("FITZ_STORAGE_LEASE_TTL_SECS", "slow")], || { + // Arrange + + // Act + let result = BootConfig::default().validate(); + + // Assert + assert!(result.is_err()); + assert!(result + .unwrap_err() + .to_string() + .contains("FITZ_STORAGE_LEASE_TTL_SECS must be an unsigned integer second count")); + }); +} + #[test] #[serial] fn should_reject_missing_required_cloud_fields_given_real_provider() { diff --git a/src/boot/storage.rs b/src/boot/storage.rs index be731a65..faf2e790 100644 --- a/src/boot/storage.rs +++ b/src/boot/storage.rs @@ -447,7 +447,9 @@ fn build_midge_open_options( // Fitz's own domain actor-reply deadlines are far shorter and will always // fire first; that is only safe because a domain timeout is answered with // a retryable error frame instead of closing the session. - let open_options = open_options.runtime_response_timeout(STORAGE_RUNTIME_RESPONSE_TIMEOUT); + let open_options = open_options + .lease_ttl(config.storage_lease_ttl()) + .runtime_response_timeout(STORAGE_RUNTIME_RESPONSE_TIMEOUT); open_options .build() diff --git a/src/testkit/transport/server.rs b/src/testkit/transport/server.rs index eaae7353..2c463ba8 100644 --- a/src/testkit/transport/server.rs +++ b/src/testkit/transport/server.rs @@ -365,6 +365,8 @@ impl TestServer { } else { crate::boot::runtime::StorageMemtableConfig::Auto }, + storage_lease_ttl_seconds: 30, + storage_lease_ttl_error: None, queue_write_policy: crate::boot::runtime::QueueWritePolicy::Fast, queue_write_policy_source: crate::boot::runtime::QueueWritePolicySource::Explicit, queue_loss_window_ms: 100,