feat: configure storage writer lease TTL

docs/development/architecture.md

@@ -829,10 +829,10 @@ seconds as a baseline, not an architectural upper bound.
 After activation, the shutdown coordinator also monitors Midge writer-lease
 renewal health. When Midge reports the lease unhealthy, Fitz withdraws
 orchestration health and strict readiness and requests fatal termination without
-attempting in-process lease reacquisition. The pinned Midge revision still needs
-an independent monotonic pre-TTL watchdog for blocked cloud renewal and
-fail-closed parsing for malformed cloud lease expiration; Fitz cannot recreate
-those lease-internal guarantees from the exposed boolean.
+attempting in-process lease reacquisition. Midge independently enforces the
+monotonic lease-valid-until deadline during provider renewal and fails closed on
+malformed cloud lease expiration. Fitz configures that deadline through
+`FITZ_STORAGE_LEASE_TTL_SECS`; Midge renews at one third of the TTL.
 ### Configuration
 **Type:** `BootConfig`
 ```rust

docs/operations/cloud-setup.md

@@ -75,6 +75,15 @@ Expect Fitz to reach readiness after storage startup and writer-lease acquisitio
 
 Any other value is rejected at startup.
 
+## Storage Writer Lease
+
+`FITZ_STORAGE_LEASE_TTL_SECS` configures the embedded Midge primary
+storage-writer lease and defaults to `30`. Values below 30 seconds are rejected.
+Midge renews the provider-backed lease at one third of the configured TTL, so
+increasing the TTL reduces coordination requests while extending takeover time
+after a crash or failed release. A graceful shutdown conditionally expires the
+lease immediately; it does not wait for the TTL.
+
 Schedule uses this policy for server-selected durable writes. KV and Stream
 still honor client-selected buffered versus sync intent, translated to
 cloud-compatible commits: buffered intent uses asynchronous cloud durability,

docs/operations/observability.md

@@ -61,12 +61,13 @@ invariant rather than deriving behavior from logs. When Midge reports the lease
 unhealthy, Fitz withdraws `/targetz`, `/readyz`, and `/healthz` and requests
 fatal broker termination; the process does not attempt in-process reacquisition.
 
-The pinned Midge revision does not yet expose or independently enforce a
-monotonic lease-valid-until deadline while a provider renewal call is blocked;
-its cloud calls can outlast the 30-second lease TTL. It also treats a malformed
-cloud lease expiration as expired. Until Midge supplies a deadline watchdog and
-fail-closed expiration parsing, do not treat provider-backed cloud takeover as
-split-brain-safe under a stalled provider or corrupt lease object.
+Midge independently enforces a monotonic lease-valid-until deadline while a
+provider renewal is in flight and treats malformed cloud lease expiration as
+indeterminate rather than expired. `FITZ_STORAGE_LEASE_TTL_SECS` configures the
+deadline, defaults to 30 seconds, and cannot be set below 30 seconds. Midge
+renews at one third of that TTL. Longer values reduce provider coordination
+traffic but extend ungraceful takeover; graceful shutdown conditionally expires
+the current lease immediately.
 
 ---
 

docs/operations/operations-runbook.md

@@ -79,11 +79,13 @@ withdraws `/targetz`, `/healthz`, and `/readyz` and terminates through the fatal
 path. Do not wait for a second signal and do not expect that process to reacquire
 the lease; verify that the orchestrator starts or promotes a healthy replacement.
 
-Provider-backed cloud failover has an upstream safety gate in the pinned Midge
-revision: renewal health has no independent monotonic deadline while provider
-I/O blocks, and malformed cloud lease expiration is treated as expired. Do not
-approve cloud takeover as split-brain-safe until Midge adds a pre-TTL watchdog
-and fail-closed expiration parsing.
+Provider-backed cloud renewal has an independent monotonic deadline in Midge,
+and malformed lease expiration fails closed as indeterminate. The storage
+writer lease TTL defaults to 30 seconds and is configured with
+`FITZ_STORAGE_LEASE_TTL_SECS`; Midge renews at one third of that value. A longer
+TTL reduces provider coordination requests but delays takeover after a crash or
+failed release. Successful graceful shutdown conditionally expires the lease
+immediately.
 
 ## Emergency Rollback
 

docs/user-guides/vars.md

@@ -130,6 +130,7 @@ For the auth and browser-perimeter checklist, see
 | FITZ_STORAGE_PREFIX | Prefix string | Unset | Optional object key namespace prefix in cloud mode. |
 | FITZ_STORAGE_CACHE_PATH | Filesystem path | ./.fitz-cloud-cache | Local cache path for cloud-backed storage mode. |
 | FITZ_STORAGE_CLOUD_DURABILITY | background or strict | background | Cloud policy for broker-selected durable writes and client sync intent. `background` completes at Midge's local cloud commit barrier and uploads asynchronously; `strict` waits for provider acknowledgement. |
+| FITZ_STORAGE_LEASE_TTL_SECS | Integer seconds, at least 30 | 30 | TTL for the embedded Midge primary storage-writer lease. Midge renews at one third of this interval. Increasing it reduces provider lease traffic but extends takeover after an ungraceful failure; graceful shutdown explicitly releases the lease. |
 | FITZ_STORAGE_MEMTABLE_BYTES | Unsigned integer byte count | Auto | Optional explicit memtable size override for embedded engine. |
 | FITZ_SCHEDULE_PRELOAD_TIMEOUT_SECS | Positive integer second count | 120 | Maximum aggregate wait for required Schedule actor preload during startup. Expiry fails startup with an explicit timeout rather than leaving the broker wedged indefinitely. |
 | FITZ_QUEUE_WRITE_POLICY | fast, buffered, or strict | fast | Queue mutation write policy. `fast` skips WAL and flushes in the background; `buffered` uses local buffered WAL or cloud asynchronous durability; `strict` waits for local sync or cloud provider acknowledgement. |

src/boot/runtime/config.rs

@@ -12,13 +12,16 @@ const DEFAULT_SQRZL_EMULATOR_BUCKET: &str = "fitz";
 const DEFAULT_METRICS_BIND_ADDR: &str = "127.0.0.1";
 const DEFAULT_METRICS_PORT: u16 = 9090;
 const ENV_STORAGE_MEMTABLE_BYTES: &str = "FITZ_STORAGE_MEMTABLE_BYTES";
+const ENV_STORAGE_LEASE_TTL_SECS: &str = "FITZ_STORAGE_LEASE_TTL_SECS";
 const ENV_QUEUE_WRITE_POLICY: &str = "FITZ_QUEUE_WRITE_POLICY";
 const ENV_QUEUE_LOSS_WINDOW_MS: &str = "FITZ_QUEUE_LOSS_WINDOW_MS";
 const ENV_KV_IDLE_TRANSACTION_TTL_SECS: &str = "FITZ_KV_IDLE_TRANSACTION_TTL_SECS";
 const ENV_SCHEDULE_PRELOAD_TIMEOUT_SECS: &str = "FITZ_SCHEDULE_PRELOAD_TIMEOUT_SECS";
 const ENV_DRAIN_GRACE_SECONDS: &str = "FITZ_DRAIN_GRACE_SECONDS";
 const ENV_DRAIN_CLOSE_REASON: &str = "FITZ_DRAIN_CLOSE_REASON";
 const DEFAULT_QUEUE_LOSS_WINDOW_MS: u64 = 100;
+const DEFAULT_STORAGE_LEASE_TTL_SECS: u64 = 30;
+const MIN_STORAGE_LEASE_TTL_SECS: u64 = 30;
 const DEFAULT_KV_IDLE_TRANSACTION_TTL_SECS: u64 = 300;
 const DEFAULT_DRAIN_GRACE_SECONDS: u64 = 25;
 const DEFAULT_DRAIN_CLOSE_REASON: &str = "broker draining for redeploy";
@@ -328,7 +331,7 @@ mod env;
 use env::{
     drain_close_reason_from_env, drain_grace_seconds_from_env, env_non_empty,
     kv_idle_transaction_ttl_seconds_from_env, queue_loss_window_ms_from_env, required_env,
-    schedule_preload_timeout_seconds_from_env,
+    schedule_preload_timeout_seconds_from_env, storage_lease_ttl_seconds_from_env,
 };
 
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -444,6 +447,15 @@ impl<'a> StorageConfig<'a> {
                 format!("{ENV_SCHEDULE_PRELOAD_TIMEOUT_SECS} must be greater than 0").into(),
             );
         }
+        if let Some(error) = &config.storage_lease_ttl_error {
+            return Err(error.clone().into());
+        }
+        if config.storage_lease_ttl_seconds < MIN_STORAGE_LEASE_TTL_SECS {
+            return Err(format!(
+                "{ENV_STORAGE_LEASE_TTL_SECS} must be at least {MIN_STORAGE_LEASE_TTL_SECS} seconds"
+            )
+            .into());
+        }
         config
             .storage_memtable
             .validate()
@@ -519,6 +531,9 @@ pub struct BootConfig {
     pub cloud_durability: CloudDurabilityMode,
     /// Optional explicit Midge memtable size in bytes.
     pub storage_memtable: StorageMemtableConfig,
+    /// TTL for the embedded Midge primary storage-writer lease.
+    pub storage_lease_ttl_seconds: u64,
+    pub(crate) storage_lease_ttl_error: Option<String>,
     /// Commit policy for queue durable mutations.
     pub queue_write_policy: QueueWritePolicy,
     /// Whether queue write policy was explicit or resolved through the default.
@@ -676,6 +691,8 @@ impl Default for BootConfig {
             kv_idle_transaction_ttl_seconds_from_env();
         let (schedule_preload_timeout_seconds, schedule_preload_timeout_error) =
             schedule_preload_timeout_seconds_from_env();
+        let (storage_lease_ttl_seconds, storage_lease_ttl_error) =
+            storage_lease_ttl_seconds_from_env();
         let (queue_write_policy, queue_write_policy_source) =
             QueueWritePolicy::from_env_with_source();
         let drain_close_reason = drain_close_reason_from_env();
@@ -704,6 +721,8 @@ impl Default for BootConfig {
             channel_capacity: 1000,
             cloud_durability: CloudDurabilityMode::from_env(),
             storage_memtable: StorageMemtableConfig::from_env(),
+            storage_lease_ttl_seconds,
+            storage_lease_ttl_error,
             queue_write_policy,
             queue_write_policy_source,
             queue_loss_window_ms,
@@ -889,6 +908,20 @@ impl BootConfig {
         self.storage_memtable.bytes()
     }
 
+    #[must_use]
+    /// Override the embedded Midge primary storage-writer lease TTL.
+    pub fn with_storage_lease_ttl_seconds(mut self, seconds: u64) -> Self {
+        self.storage_lease_ttl_seconds = seconds;
+        self.storage_lease_ttl_error = None;
+        self
+    }
+
+    #[must_use]
+    /// Return the configured embedded Midge primary storage-writer lease TTL.
+    pub fn storage_lease_ttl(&self) -> Duration {
+        Duration::from_secs(self.storage_lease_ttl_seconds)
+    }
+
     /// Validate the full broker boot configuration.
     ///
     /// # Errors

src/boot/runtime/config/env.rs

@@ -1,7 +1,8 @@
 use super::{
     DEFAULT_DRAIN_CLOSE_REASON, DEFAULT_DRAIN_GRACE_SECONDS, DEFAULT_KV_IDLE_TRANSACTION_TTL_SECS,
-    DEFAULT_QUEUE_LOSS_WINDOW_MS, ENV_DRAIN_CLOSE_REASON, ENV_DRAIN_GRACE_SECONDS,
-    ENV_KV_IDLE_TRANSACTION_TTL_SECS, ENV_QUEUE_LOSS_WINDOW_MS, ENV_SCHEDULE_PRELOAD_TIMEOUT_SECS,
+    DEFAULT_QUEUE_LOSS_WINDOW_MS, DEFAULT_STORAGE_LEASE_TTL_SECS, ENV_DRAIN_CLOSE_REASON,
+    ENV_DRAIN_GRACE_SECONDS, ENV_KV_IDLE_TRANSACTION_TTL_SECS, ENV_QUEUE_LOSS_WINDOW_MS,
+    ENV_SCHEDULE_PRELOAD_TIMEOUT_SECS, ENV_STORAGE_LEASE_TTL_SECS,
 };
 
 pub(super) fn env_non_empty(key: &str) -> Option<String> {
@@ -92,6 +93,14 @@ pub(super) fn schedule_preload_timeout_seconds_from_env() -> (u64, Option<String
     )
 }
 
+pub(super) fn storage_lease_ttl_seconds_from_env() -> (u64, Option<String>) {
+    positive_u64_from_env(
+        ENV_STORAGE_LEASE_TTL_SECS,
+        DEFAULT_STORAGE_LEASE_TTL_SECS,
+        "second count",
+    )
+}
+
 pub(super) fn drain_close_reason_from_env() -> String {
     env_non_empty(ENV_DRAIN_CLOSE_REASON).unwrap_or_else(|| DEFAULT_DRAIN_CLOSE_REASON.to_string())
 }

src/boot/runtime/config/tests/base_auth_and_network.rs

@@ -94,6 +94,7 @@ pub(super) fn with_storage_env<T>(values: &[(&str, &str)], test: impl FnOnce() -
         "FITZ_STORAGE_NAMESPACE",
         "FITZ_STORAGE_ACCOUNT",
         "FITZ_STORAGE_CLOUD_DURABILITY",
+        "FITZ_STORAGE_LEASE_TTL_SECS",
         "FITZ_STORAGE_MEMTABLE_BYTES",
         ENV_QUEUE_WRITE_POLICY,
         ENV_QUEUE_LOSS_WINDOW_MS,

src/boot/runtime/config/tests/cloud_storage.rs

@@ -1,5 +1,70 @@
 use super::*;
 
+#[test]
+#[serial]
+fn should_default_storage_lease_ttl_to_thirty_seconds() {
+    with_storage_env(&[], || {
+        // Arrange
+
+        // Act
+        let config = BootConfig::default();
+
+        // Assert
+        assert_eq!(config.storage_lease_ttl(), Duration::from_secs(30));
+    });
+}
+
+#[test]
+#[serial]
+fn should_read_storage_lease_ttl_from_environment() {
+    with_storage_env(&[("FITZ_STORAGE_LEASE_TTL_SECS", "59")], || {
+        // Arrange
+
+        // Act
+        let config = BootConfig::default().with_auth_config(crate::auth::AuthConfig::Disabled);
+
+        // Assert
+        assert_eq!(config.storage_lease_ttl(), Duration::from_secs(59));
+        assert!(config.validate().is_ok());
+    });
+}
+
+#[test]
+#[serial]
+fn should_reject_storage_lease_ttl_below_current_safety_floor() {
+    with_storage_env(&[("FITZ_STORAGE_LEASE_TTL_SECS", "29")], || {
+        // Arrange
+
+        // Act
+        let result = BootConfig::default().validate();
+
+        // Assert
+        assert!(result.is_err());
+        assert!(result
+            .unwrap_err()
+            .to_string()
+            .contains("FITZ_STORAGE_LEASE_TTL_SECS must be at least 30 seconds"));
+    });
+}
+
+#[test]
+#[serial]
+fn should_reject_non_numeric_storage_lease_ttl() {
+    with_storage_env(&[("FITZ_STORAGE_LEASE_TTL_SECS", "slow")], || {
+        // Arrange
+
+        // Act
+        let result = BootConfig::default().validate();
+
+        // Assert
+        assert!(result.is_err());
+        assert!(result
+            .unwrap_err()
+            .to_string()
+            .contains("FITZ_STORAGE_LEASE_TTL_SECS must be an unsigned integer second count"));
+    });
+}
+
 #[test]
 #[serial]
 fn should_reject_missing_required_cloud_fields_given_real_provider() {

src/boot/storage.rs

@@ -447,7 +447,9 @@ fn build_midge_open_options(
     // Fitz's own domain actor-reply deadlines are far shorter and will always
     // fire first; that is only safe because a domain timeout is answered with
     // a retryable error frame instead of closing the session.
-    let open_options = open_options.runtime_response_timeout(STORAGE_RUNTIME_RESPONSE_TIMEOUT);
+    let open_options = open_options
+        .lease_ttl(config.storage_lease_ttl())
+        .runtime_response_timeout(STORAGE_RUNTIME_RESPONSE_TIMEOUT);
 
     open_options
         .build()

src/testkit/transport/server.rs

@@ -365,6 +365,8 @@ impl TestServer {
             } else {
                 crate::boot::runtime::StorageMemtableConfig::Auto
             },
+            storage_lease_ttl_seconds: 30,
+            storage_lease_ttl_error: None,
             queue_write_policy: crate::boot::runtime::QueueWritePolicy::Fast,
             queue_write_policy_source: crate::boot::runtime::QueueWritePolicySource::Explicit,
             queue_loss_window_ms: 100,