RFC: AWX, Temporal, and the operation execution boundary

[znerol2]

AWX, Temporal, and the operation execution boundary

Status: Proposal for discussion. This RFC does not amend an accepted ADR,
reopen a closed issue, or authorize a backend migration.

Decision question

Should dmfdeploy keep AWX as its operation actuator, adopt Temporal for durable
workflow orchestration, or use a smaller Kubernetes-native runner for specific
work? The decision must serve the shared-console and media-workload roadmap,
including deploys that do not block the facility.

Current position and tension

AWX is the accepted catalog actuator. ADR-0013, ADR-0025, ADR-0037, and
architectural-commitments-v1 bind the current path: catalog intent → AWX job
template → Ansible launcher → Helm/runtime and NetBox lifecycle state. ADR-0043
limits scale-to-zero availability to AWX. The v0.1 freeze names the
"Argo-for-k8s + AWX-glue hybrid actuator" as a non-goal; removing that freeze
requires a dated amendment. That historical hybrid proposal does not make
Argo CD and Argo Workflows interchangeable.

The later work order says AWX gets operational fixes only. Lessons from
#309/#327/#492, closed as sunk cost, feed the active #760 replacement
evaluation. Its criteria cover retryable dependency failures, lock recovery,
priority on a saturated node, restart survival, structured outcomes, runtime
secrets, and cheap wake/sleep. The evaluation recommends whether to move; an
ADR follows an RFC.

The work order says AWX sleeps only in the sandbox demo lane; the closed
#309/#327/#492 failures were not evidence that normal deployments are broken.
This RFC does not silently reverse the Temporal direction or claim a migration
is decided. This RFC proposes evaluating Temporal now under #760, without making
v0.2 feature delivery wait for that decision.

Options

1. Bounded AWX operations. Keep AWX for job templates and Ansible, preserving
   the proven catalog path, execution environments, credentials, job events,
   and audit integration. It retains cold-start and internal failure modes in
   the sandbox profile. The proposed renewable-hold work in closed issue AWX autoscale: REST-only plays (693) need an explicit renewable Lease hold — activity_stream cannot see admin-driven or idempotent API work #304
   was classified as sunk cost. Issue AWX autoscale: REST-only plays (693) need an explicit renewable Lease hold — activity_stream cannot see admin-driven or idempotent API work #304 remains closed; an interim hold
   needs a documented operational justification before merge or rollout.
2. Temporal orchestration. Put long-lived workflow state, retry policy,
   timers, and recovery in Temporal; execute side effects in Activities. A
   worker may run the existing Ansible launcher in a Kubernetes pod. This can
   remove AWX from the execution path only if the project deliberately replaces
   the job-template, credential, execution, event, and audit functions callers
   use today. Calling AWX from a Temporal Activity retains AWX and adds a
   control plane.
3. Argo Workflows. Consider for finite Kubernetes pod/DAG orchestration
   where Kubernetes Workflow objects, retries, and synchronization locks fit
   the task (Argo architecture).
   Argo Workflows is distinct from Argo CD. Using it as the catalog actuator
   requires an actuator ADR amendment. The frozen Argo-for-Kubernetes +
   AWX-glue hybrid is a separate choice and requires its own dated amendment.
4. Kubernetes Job plus domain state. Use a Job or CronJob for a bounded
   task and let the owning application persist its operation state and
   idempotency keys. This is the smallest comparator for finite Ansible work;
   a Job by itself does not provide durable business-level resume, outcome
   history, or compensation.

Proposed evaluation boundary

Keep request, operation ID, structured outcome, authorization, and audit
context independent of the runner. The AWX adapter maps requests to templates
and events/status back to operations; any replacement preserves that contract.
NetBox remains instance inventory, placement, and lifecycle-intent truth;
NMOS remains live connection and flow-state truth. Workflow requests, results,
logs, and history contain references and sanitized outcomes; secrets are
fetched at run time and never persisted in payloads.

Use #760's criteria plus cancellation, lost-response idempotency, and one
representative lifecycle vertical. Compare the simplest viable Kubernetes Job,
AWX, and Temporal-worker paths for resource use, recovery, audit, operator
steps, and implementation surface. A successful prototype informs a separate
adoption decision.

Temporal retries Activities, so side effects must be idempotent (Temporal
Tasks). Self-hosting adds a server and
persistence database (persistence),
plus security, monitoring, and upgrade work. Production guidance recommends
two workers on separate hosts for always-on queues; scale-to-zero adds queue
delay (worker deployment).
Test those costs against dmfdeploy's single-node proof profile and small-team
operating model.

The draft AWX job-hold PR #117 overlaps the closed #304 problem
class. Its 122 local tests pass, but it has no recorded live evidence. Keep it
draft while #760 establishes whether this need justifies new AWX-specific
investment. This RFC does not request merging or expanding it, and does not
change #304's closed state.

Proposed questions for discussion

1. Does Evaluate a replacement for AWX against the failures seen so far #760's acceptance set describe the real migration need, or is a
   required lifecycle/cancellation/audit case missing?
2. Should the first candidate run Ansible directly in a Temporal Worker pod,
   or should a durable workflow call AWX during a transition period?
3. Which workload and failure injection provide a fair single-node proof
   without making v0.2 delivery depend on the engine decision?
4. What measured threshold would justify operating Temporal's additional
   database, service, and worker fleet?
5. What operational evidence would justify an interim AWX hold while Evaluate a replacement for AWX against the failures seen so far #760
   proceeds, and is that sufficient to advance draft PR docs(plans): nmos_crosspoint media function plan #117 under the current
   work order?

References

• Issue #760: Evaluate a replacement for AWX against the failures seen so far
• Issue #304: REST-only AWX plays need a renewable hold
• PR #117: AWX job-hold implementation (draft)
• Issue #669: Closed-loop assurance pattern
• Platform debt work order
• Discussion #112, Topic 6: AWX / Argo CD / Temporal split
• ADR-0013: Media Function catalog model
• ADR-0025: Ansible in in-cluster pods and catalog Helm
• ADR-0037: NetBox-driven Media Function inventory
• ADR-0043: AWX scale-to-zero availability
• Temporal self-hosted production checklist
• Temporal persistence
• Temporal worker deployment
• Argo Workflows architecture
• Argo Workflows synchronization
• AWX Operator