Skip to content

Handle late ProtocolDAGResult uploads from compute services with lost registrations consistently #513

Description

@dotsdl

A compute service whose ComputeServiceRegistration has been lost (e.g. expired during a network partition longer than the expiry window; its heartbeats then no-op silently and nothing re-registers it) can still upload ProtocolDAGResults: compute API auth is identity-based, and the results route performs no claim or registration check. Accepting these late results is deliberate — completed work should never be discarded — and with the v0.8.0 introspection design, the provenance side is handled consistently (TaskProvenance finalization matches by (task, compute_service_id); a late result finalizes its own attempt record, overwriting expiredcomplete/error).

Task status handling on late results is not consistent, however:

  1. Task back at waiting (expired, not yet reclaimed): the results route stores the ProtocolDAGResultRef and calls set_task_complete, but waiting → complete is a disallowed transition, so it no-ops. Result: the state store holds a successful result for the Task (and, from 0.8.0, a provenance record saying the attempt completed), yet the Task remains waiting and will be claimed and fully re-run — wasted compute and an internally inconsistent picture.

  2. Task reclaimed and running under another service: the late success flips the Task to complete and strips the new claimant's CLAIMS relationship out from under it; the new claimant's eventual result lands as an additional ProtocolDAGResultRef on a completed Task (tolerated, since complete → complete is allowed).

Proposal:

  • Allow the server-side late-success path to take waiting → complete (scoped to the results route, not user-facing status setting), so a stored valid success completes the Task instead of letting it re-run. The corresponding TaskHub ACTIONS/APPLIES cleanup that set_task_complete performs should apply as usual.
  • Decide and document the analogous behavior for late failed results on a waiting Task (currently set_task_error also no-ops from waiting; leaving the Task waiting is probably correct — it re-runs, which is what an error + restart policy would produce — but this should be an explicit decision).
  • Keep and document case 2 as "first valid success wins" (or revisit if we prefer to protect the active claimant).

Related: the v0.8.0 introspection design (execution provenance / TaskProvenance finalization semantics), #106, #211.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions