fix(substrate): drain warm actors on shutdown to prevent RUNNING leak - #5
Merged
Merged
Conversation
warm-then-suspend defers each actor's suspension to a time.AfterFunc timer that lives only in ax-server memory. When ax-server restarts or redeploys before a timer fires, the timer dies with the process and the actor is never suspended -- it leaks as a RUNNING actor holding a worker slot and memory until manually reclaimed. This is the reclaim prerequisite google#284's spec listed for production enablement. Add an optional harness.Drainer capability (Shutdown). SubstrateHarness implements it by stopping every pending idle timer and suspending the warm actor synchronously; in-turn actors are left to their own Close, addr-less entries are dropped, and in-progress timer suspensions are awaited. Registry .Close drains every registered Drainer, and Server.GracefulStop now drains the gRPC server before closing the controller so in-flight turns finish (and park their actors) before the drain runs -- otherwise a turn completing after the drain would re-arm a leaking timer. immediate-suspend mode is unaffected (no warm actors, Shutdown is a no-op). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
warm-then-suspend(AgentFleet google#213 fix) defers each actor's suspension to atime.AfterFunctimer that lives only in ax-server process memory. When ax-server restarts or redeploys before a timer fires, the timer dies with the process and the actor is never suspended — it leaks as aRUNNINGactor holding a worker slot + memory until manually reclaimed. This compounds the actor-leak / atelet-OOM family (agentfleet google#210/google#211/google#310) and is the reclaim prerequisite google#284's spec listed for production enablement.Exposure is currently small (
AX_SUBSTRATE_IDLE_TIMEOUT=60s+ax-server replicas=1), but every ax-server restart (including OOM-driven ones) is a leak opportunity.Fix
harness.Drainer— new optional capability:Shutdown(ctx)releases per-conversation resources that outlive individual turns.SubstrateHarness.Shutdown— stops every pending idle timer and suspends the warm actor synchronously. In-turn actors are left to their ownClose; addr-less bookkeeping entries are dropped; in-progress timer-driven suspensions are awaited. Generation is bumped so a timer callback that already fired but is still blocked onidleMubecomes a no-op (no double-suspend). No-op inimmediate-suspendmode.Registry.Closedrains every registeredDrainer.Server.GracefulStopnow drains the gRPC server before closing the controller, so in-flight turns finish (and park their actors for idle suspension) before the drain runs — otherwise a turn completing after the drain would re-arm a leaking timer.Tests (TDD)
TestSubstrateHarness_ShutdownDrainsWarmActorsAwaitingIdleSuspend— warm actor with a 1h pending timer is suspended onShutdown, entry removed.TestSubstrateHarness_ShutdownLeavesActiveTurnAlone— in-turn actor untouched.TestSubstrateHarness_ShutdownImmediateModeIsNoOp.TestRegistry_CloseDrainsDrainerHarnesses— non-Drainer harness skipped without error.Full module suite green;
-raceclean on substrate/controller/server;go vet+ gofmt clean.🤖 Generated with Claude Code