Skip to content

feat(ext-tasks): durable storage, retention, and multi-instance task resumption - #19

Merged
hortovanyi merged 1 commit into
mainfrom
feat/ext-tasks-durable-storage
Aug 15, 2026
Merged

feat(ext-tasks): durable storage, retention, and multi-instance task resumption#19
hortovanyi merged 1 commit into
mainfrom
feat/ext-tasks-durable-storage

Conversation

@hortovanyi

Copy link
Copy Markdown
Contributor

Closes #86, #88, #89 (internal task list).

What

The 2026-07-28 Tasks extension shipped with an in-memory store only, while the
superseded 2025 crate carried SQLite/Postgres/DynamoDB. That is now inverted:
turul-mcp-ext-tasks owns task persistence and carries all four backends, laid out
like the sibling storage crates (traits.rs + one module per backend).

Every status rule, owner check and tasks/update key decision lives once in
traits.rs as pure transitions. A backend only does load → apply → store under its
own atomicity boundary — a row lock for SQL, optimistic concurrency (rev +
ConditionExpression) for DynamoDB, which has no read-modify-write transaction. That
makes cross-backend parity structural rather than hoped for, and it is enforced by a
14-invariant contract run against all four.

Behavioural changes

  • Multi-instance resumption. A parked worker waited on an in-process oneshot, so a
    tasks/update landing on any other instance was silently lost and the task sat in
    input_required forever — precisely the deployment a shared store exists for. The
    worker now watches the store, the same store-and-fetch shape SessionStorage already
    uses for SSE events. Pinned by a two-server e2e that fails without it.
  • Retention — orphan recovery, per-task ttlMs, terminal deletion, plus DynamoDB's
    native ttlEpoch. One with_ext_tasks_retention call configures both the sweep loop
    and the per-item TTL writer. Without it nothing sweeps, which is the correct default
    and now a testable one.
  • MRTR-before-task composition (SEP-2663 §Composition) via
    ExtTaskElection::with_mrtr_first.
  • Client fix: call_tool_or_task_with_input_responses. The existing MRTR retry
    routed through the strict resultType check, which rejects "task" — so no client
    could consume the composition at all
    . Found by the e2e: server, wire and spec were
    each correct and the pair was still broken.

Provenance

The SEP prose is vendored and checksummed alongside the schema. The rules that govern
sequencing exist only in prose, so pinning the schema alone left them resting on an
unpinned web page. Verified against the published extension overview; it confirms the
tasks/update inert-key decision verbatim ("Ignore responses for unknown or
already-satisfied keys") and surfaced one untested case, now covered.

Gating

scripts/ext-tasks-backends.sh owns its services — probes Postgres, fetches and starts
DynamoDB Local (jar, no docker) — and fails rather than skips when one is
unreachable. turul-mcp-task-storage is deprecated with an owner and a removal trigger.

Verification

  • Full gates: 79 PASS / 0 FAIL
  • Upstream conformance: 37/37 scored
  • 58 ext-tasks tests — 22 client e2e (4 against real SQLite/Postgres/DynamoDB),
    15 server wire, 21 crate

Every behavioural claim was proven by breaking the code and watching the test fail.

Versions

turul-mcp-ext-tasks 0.1.3 · turul-mcp-client 0.4.1 · turul-mcp-server 0.4.4

Breaking for out-of-workspace TaskStore implementors: sweep is a new required
method, provide_input takes an untyped map, and TaskStoreError::UnknownInputKey is
replaced by InvalidInputResponse { key, detail }. Patch bumps per
docs/rules/crate-versioning.md; flagged because 0.1.x gives no other signal.

…resumption

The 2026-07-28 Tasks extension shipped with an in-memory store only, while the
superseded 2025 crate carried SQLite/Postgres/DynamoDB. That is now inverted:
turul-mcp-ext-tasks owns task persistence and carries all four backends, laid
out like the sibling storage crates (traits.rs + one module per backend).

Every status rule, owner check and tasks/update key decision lives once in
traits.rs as pure transitions; a backend only does load -> apply -> store under
its own atomicity boundary (a row lock for SQL, optimistic concurrency for
DynamoDB, which has no read-modify-write transaction). Cross-backend parity is
therefore structural, and enforced by a 14-invariant contract run against all
four — the check the 2025 crate had in design and never wired up.

Behavioural changes:

- Multi-instance resumption. A parked worker waited on an in-process oneshot,
  so a tasks/update landing on any other instance was silently lost and the
  task sat in input_required forever — precisely the deployment a shared store
  is for. The worker now watches the store, the same store-and-fetch shape
  SessionStorage already uses for SSE events. Pinned by a two-server e2e.

- Retention: orphan recovery, per-task ttlMs, terminal deletion, plus
  DynamoDB's native ttlEpoch. One with_ext_tasks_retention call configures the
  sweep loop AND the per-item TTL writer; without it nothing sweeps, which is
  the correct default and now a testable one.

- MRTR-before-task composition (SEP-2663 §Composition), reachable via
  ExtTaskElection::with_mrtr_first.

- Client: call_tool_or_task_with_input_responses. The existing MRTR retry
  routed through the strict resultType check, which rejects "task", so no
  client could consume the composition at all. Found by the e2e — server, wire
  and spec were each correct and the pair was still broken.

The SEP prose is now vendored and checksummed alongside the schema: the rules
that govern sequencing exist only in prose, so pinning the schema alone left
them resting on an unpinned web page.

Backends are gated by scripts/ext-tasks-backends.sh, which owns its services —
probes Postgres, fetches and starts DynamoDB Local (jar, no docker) — and FAILS
rather than skips when one is unreachable. turul-mcp-task-storage is deprecated
with an owner and a removal trigger.

Versions: turul-mcp-ext-tasks 0.1.3, turul-mcp-client 0.4.1, turul-mcp-server
0.4.4. Breaking for out-of-workspace TaskStore implementors: sweep is a new
required method and provide_input takes an untyped map.

Gates: 79 PASS / 0 FAIL, conformance 37/37 scored.
@hortovanyi
hortovanyi merged commit 49dfbce into main Aug 15, 2026
8 of 10 checks passed
@hortovanyi
hortovanyi deleted the feat/ext-tasks-durable-storage branch August 15, 2026 22:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant