Capture compute execution environment in Task provenance (#106)

[dotsdl]

Summary

Records the software environment each Task execution attempt ran in, as part of durable execution provenance — addressing this request on #106:

Perhaps on top of this, we can also attempt to get full environment information via a sequence of attempts to call micromamba, mamba, conda, and pip. Some information on the environment used for Task execution would be better than no information, and we could try to structure this in a usable way.

Builds on #514 (the introspection/provenance work); this PR is based on feature/task-introspection-0.8.0 and will retarget to main once #514 merges.

What it does

• Capture (compute/environment.py): a best-effort capture_environment() tries micromamba → mamba → conda → pip (each --json), takes the first that yields a parseable package listing, and returns {"tool": ..., "packages": {name: version}, "captured_at": ...}. Never raises; a service with no package manager simply records no environment. Gated by ComputeServiceSettings.capture_environment (default True), captured once at startup (the env is fixed for the service's lifetime) and sent at registration.
• Storage — a content-addressed ComputeEnvironment node (hash of tool + package map), MERGE'd at registration so identical environments across services and claims are stored once. Each claim links its TaskProvenance attempt to the environment via RAN_IN. The node deliberately outlives the ComputeServiceRegistration, so an attempt's environment survives service teardown (expiry/deregistration) — exactly the case provenance exists to record.
• Surface — TaskAttempt.environment (from get_task_history and the most-recent-attempt in get_tasks_details) carries the captured {tool, packages, captured_at}.
• Migration v07_to_v08 — adds the ComputeEnvironment.hash uniqueness constraint. Unlike the (removed) TaskProvenance index, this one is load-bearing: it makes the dedup MERGE correct under concurrency and keeps Neo4jStore.check consistent on upgrade. Idempotent; no data migration.

Design notes

• Why a dedup node, not a per-attempt blob: a package listing is ~10–20 KB and is identical across all of a service's claims (and often across services on the same image). Copying it onto every TaskProvenance node (like hostname) would undercut the "provenance nodes are tiny" property; the content-addressed node stores each distinct environment once.
• Why this migration is justified (when the earlier one wasn't): the TaskProvenance label indexes were dropped because every provenance query is Task-anchored and never scans the label. Here the constraint backs a MERGE on ComputeEnvironment.hash — a genuine keyed lookup — so it earns its keep.

Testing

• Unit (runnable locally, all green): capture_environment fallback/parse/failure paths (mocked subprocess), and ComputeEnvironment model round-trips (order-independent hash, from_capture/from_node/to_capture_dict), plus TaskAttempt.environment.
• Integration (Neo4j, in CI): environment surfaced on get_task_history; deduplication across services (same env → one node, different env → two); and survival of registration expiry. Docs (compute.rst, introspection.rst, operations.rst) and the #106 news fragment updated.

🤖 Generated with Claude Code

[dotsdl]

Code review

Reviewed the environment-capture work (diff scoped against feature/task-introspection-0.8.0). Overall this is clean and well-tested — cross-file wiring is consistent, all new fields are optional/backwards-compatible, and the dedup/survival semantics are correctly exercised by the integration tests. No crash- or corruption-level bug found. Three findings worth a look, most-significant first.

Findings

1. Deduplicated captured_at is the first service's timestamp, but is surfaced and documented as per-attempt — storage/statestore.py:1579, storage/models.py:326

content_hash covers only (tool, packages), and the MERGE ... ON CREATE SET ce.captured_at = $captured_at sets the timestamp only when the node is first created. So when a second service captures an identical package set at a later time, its attempts link to the pre-existing node bearing the first service's captured_at.

Scenario: Service A (started Monday) and Service B (started Friday) run the same image. A Task run by B on Friday reports environment["captured_at"] = Monday — a time before B existed. Both the docs (introspection.rst: "captured by the compute service at startup") and the ComputeEnvironment.captured_at docstring ("When the environment was captured on the compute service") assert a per-service meaning the deduplicated field can't hold. Consider dropping captured_at from the surfaced node, relabeling it "first observed," or excluding it from the client-facing dict.

2. Environment capture runs up to four 60s subprocesses synchronously in __init__, before registration — compute/service.py:89

capture_environment() is called inline in SynchronousComputeService.__init__, with a 60s timeout per tool. A hanging or very slow package manager delays service startup (and therefore task claiming) — worst case ~240s if early tools are present but hang and only pip succeeds.

Scenario: On an HPC node where conda list stalls on a locked/NFS-backed env, the service blocks at startup until the timeout, appearing hung to schedulers/liveness probes. It's gated by capture_environment=False, but a shorter per-tool timeout (or a note in the setting's docs) would harden this.

3. A malformed environment payload aborts the whole registration — storage/models.py:332 (from_capture)

register_computeservice calls ComputeEnvironment.from_capture(environment) inside the registration transaction; from_capture does unguarded environment["tool"] / environment["packages"]. A dict missing those keys raises KeyError, rolling back the entire registration (500), not just skipping the environment.

Scenario: An authenticated compute client (or a future capture path) sends {"captured_at": "..."} without packages → registration fails outright. Low severity (the normal capture_environment path never produces this, and it's self-inflicted), but a defensive skip-on-malformed would keep registration robust.

Checked and cleared (considered, but not real)

• Cypher ce "out of scope" in CLAIM_QUERY — there is no WITH between OPTIONAL MATCH (csreg)-[:HAS_ENVIRONMENT]->(ce) and the FOREACH, so ce stays bound across the intervening CREATEs; test_environment_surfaced_on_task_history confirms RAN_IN is created. Not a bug.
• "Store packages as a native map instead of a JSON string" — Neo4j node properties can't hold nested maps, so the json.dumps is required (as the inline comment notes).
• from_node JSONDecodeError on corrupted packages — only triggerable by out-of-band tampering of data this code itself writes; not actionable.
• Redundant str()/json micro-coercions and a "generic dedup helper" — trivial/speculative.

_{🤖 Generated with Claude Code}