test: isolate cloud recovery lifecycle timing

[smiggleworth]

Summary

Fix the two independent test races reported by consecutive Core attempts on final main after PR #267.

• Keep the injected 100 ms cloud shutdown drain timeout confined to the intentionally failing upload lifecycle; healthy recovery engines now use the normal timeout while reopening the same database.
• End fake hydration requests at the callback completion boundary before notifying the recovery caller, preventing already-completed requests from overlapping the next bounded batch in test instrumentation.

Production recovery, persistence, deadline, and batching behavior is unchanged.

Why are you making this contribution?

Final-main CI run https://github.com/cntryl/midge/actions/runs/33527456724 failed twice with different test-only races. Attempt 1 timed out a healthy recovery shutdown because the intentionally short failure timeout leaked into the reopened engine. Attempt 2 recorded more than eight hydration requests because the fake backend decremented its counter after the callback had already admitted the next batch.

Linked issues

No issue: this is a test-only follow-up to consecutive red final-main Core attempts after PR #267.

Acceptance audit

• Criterion: Cloud upload-failure tests use the short shutdown timeout only for the intentionally failing lifecycle, and bounded-hydration instrumentation measures requests through completion without overlapping the next batch.
  Evidence: The full all-features workspace suite passes; the three upload-failure lifecycles passed 5 of 5 focused repetitions each; bounded hydration passed 25 of 25 focused repetitions; its timeout sibling passed 5 of 5.
  Production entry point: Cloud shutdown recovery in tests/cloud_persistence_hardening.rs and bounded cloud WAL hydration in src/engine/startup/cloud_recovery.rs.
  Resolution: Separate failure and recovery options over the same database path, and decrement the fake backend in-flight counter before publishing completion.

Risk and compatibility

Low risk. This PR changes test setup and instrumentation only. It does not change production code, persistence formats, durability behavior, cloud protocols, recovery policy, or public APIs.

Verification

• cargo test --workspace --all-features
• cargo test --all-features --test cloud_persistence_hardening -- --nocapture: 17 passed
• Formerly failing upload recovery test: 5 of 5 focused repetitions passed
• Two sibling upload-failure lifecycle tests: 5 of 5 focused repetitions each passed
• Bounded cloud WAL hydration: 25 of 25 focused repetitions passed
• Cloud WAL hydration timeout sibling: 5 of 5 focused repetitions passed
• cargo clippy --workspace --all-targets --all-features -- -D warnings -D clippy::pedantic
• cargo fmt --check
• cntryl-tools validate-tests: 2579 of 2579 tests compliant
• git diff --cached --check

Tool assistance disclosure

• AI or another generative tool materially assisted this contribution.

Codex inspected both failed hosted attempts, traced each failure to test lifecycle or instrumentation timing, audited sibling tests, implemented the test-only fixes, and ran the reported verification.

Contributor responsibility

• I understand the complete change and can explain or revise it.
• I reviewed the complete diff.
• I reported validation accurately and did not claim checks I did not run.
• I disclosed material generated assistance.
• I have the right to submit this work under the repository license.

[Copilot]

Pull request overview

This PR tightens test-only lifecycle and instrumentation timing around cloud recovery to eliminate two independent races seen in consecutive final main Core CI attempts, without changing production recovery or persistence behavior.

Changes:

• Split cloud test options so the short (100ms) cloud shutdown drain timeout is confined to the intentionally failing upload lifecycle; healthy reopen/recovery uses normal timeouts.
• Adjust the fake cloud backend’s “in-flight” instrumentation to end the measured interval before sending the completion callback, preventing overlap with the next bounded batch.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
tests/cloud_persistence_hardening.rs	Separates “failure” vs “recovery” open options so the short drain timeout doesn’t leak into the healthy reopen path.
src/engine/startup/tests.rs	Moves in_flight decrement to occur before publishing the callback to avoid batch-overlap in bounded hydration tests.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.