fix(tasks): reap stale running tasks on startup + auto-purge loop - #6
Open
catoncat wants to merge 1 commit into
Open
fix(tasks): reap stale running tasks on startup + auto-purge loop#6catoncat wants to merge 1 commit into
catoncat wants to merge 1 commit into
Conversation
Two persistent issues with the task store are addressed: 1. Zombie 'running' status. Worker threads are daemon=True with no try/finally guarantee that meta.json is updated on abrupt exit (launchd kill, OOM, supervisor reload, uncaught exception). Once that happens the task's status sticks at 'running' forever and get_task / wait_task lie to consumers. We now run TaskStore.reap_stale_running_tasks() on every server startup, flipping persisted queued/running entries to 'abandoned' with a reason field. 'abandoned' is added to TERMINAL_TASK_STATUSES so wait_task treats it as a terminal state. 2. No automatic disk cleanup. purge_tasks was only an RPC tool, never scheduled. We now spawn a daemon thread on startup that calls store.purge_tasks(...) on a configurable interval (env NOTION_LOCAL_OPS_AUTO_PURGE_INTERVAL_SECONDS, default 3600s) using a configurable retention window (env NOTION_LOCAL_OPS_AUTO_PURGE_OLDER_HOURS, default 168h = 7 days). The MCP purge_tasks tool also gains an optional 'statuses' filter so callers can narrow purges to e.g. cancelled/failed/abandoned. No behavior changes for the happy path: successful tasks still report 'succeeded' immediately and are purged on the same age basis.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Two persistent issues with the on-disk task store, surfaced again today after a server restart:
runningstatus. Worker threads inExecutorRegistryaredaemon=Trueand have notry/finallyguard around themeta.jsonwrite. If the MCP process exits abruptly (launchd kill, OOM, supervisor reload, uncaught exception), the task's status sticks atrunningforever.get_task/wait_taskthen lie to consumers indefinitely.purge_taskswas only an RPC tool, never scheduled. The task directory underSTATE_DIR/tasksgrew unbounded — today there were 38 stale dirs purged manually plus 30 more sitting around, including 2 that had been "running" for ~5 days but the actual subprocess was long dead.What
tasks.pyTaskStore.reap_stale_running_tasks(reason=...)flips every persistedqueued/runningentry to"abandoned"with anabandoned_reasonfield and bumpedupdated_at. Idempotent and lock-protected.purge_tasksgains an optionalstatuseswhitelist so callers can narrow purges to e.g.cancelled/failed/abandoned. When unset, behavior is identical to before.NON_TERMINAL_STATUSES = {"queued", "running"}constant for clarity.executors.pyTERMINAL_TASK_STATUSESnow includes"abandoned"sowait_task/get_tasktreat reaped tasks as terminal.server.pymain()now callsstore.reap_stale_running_tasks(reason="server_startup")once at startup, before binding uvicorn. Logged.notion-local-ops-auto-purgerunsstore.purge_tasks(...)on a configurable interval. Knobs:NOTION_LOCAL_OPS_AUTO_PURGE_INTERVAL_SECONDS(default3600)NOTION_LOCAL_OPS_AUTO_PURGE_OLDER_HOURS(default168, i.e. 7 days)<= 0to disable.server_infonow reports auto-purge state (interval_seconds,older_than_hours,running).purge_tasks(statuses=...)argument exposed on the MCP tool.reap_stale_tasksMCP tool for manual recovery (also called automatically on startup).Compatibility
succeeded/failed/cancelledflow exactly as before; same age-based purge by default.statusother thanqueued/runningare untouched by reap._terminate_processbefore reload, so flipping them toabandonedis correct.Manual smoke test plan
run_command_streamwith a long-running command, thenkill -9the MCP pid.get_task(<id>)should reportstatus="abandoned",abandoned_reason="server_startup",completed=true. Previously this hung atrunning.NOTION_LOCAL_OPS_AUTO_PURGE_INTERVAL_SECONDS=10for fast verification), the task dir disappears once it crosses the age threshold.purge_tasks(statuses=["abandoned"], older_than_hours=0)deletes only abandoned tasks immediately.Future work (out of scope for this PR)