This is a rough sketch of the idea. I never worked with ASF, so raising this mostly to gather feedback and highlight a possibility.
Idea
Compile a WIO program to an AWS Step Functions state machine (ASL JSON). The Scala app is deployed as a single "fat" Lambda that hosts a dispatch table for every Scala function in the WIO tree. ASL handles control flow and durability; the Lambda handles all computation.
High-level shape
The compiler walks a WIO and produces:
- ASL JSON — state machine structure (transitions, Choice, Parallel, Wait, Catch, Retry).
- A dispatch registry
Map[TaskId, ...] — every WIO node holding a Scala function gets a stable ID and a slot. This covers RunIO, Pure, Transform, Fork predicates, dynamic Timer durations, Loop conditions, Parallel combiners — effectively every node that isn't pure structure.
At runtime, every ASL Task state invokes the same Lambda with {taskId, state}. The Lambda looks up taskId, runs the Scala function, and returns a new state.
Node mapping
(Approximate, to be verified)
RunIO / Pure / Transform → Task (Lambda dispatch)AndThen → NextFork → Task computing a discriminator, then Choice on it. (ASL Choice is restricted comparisons over JSONPath and can't express arbitrary Scala predicates; routing through the Lambda keeps this general.)Loop → body subgraph + Choice + back-edgeParallel → ASL Parallel with a Lambda combinerHandleError / HandleErrorWith → CatchRetry → ASL Retry directivesTimer static → Wait; dynamic → Lambda computes duration → Wait with SecondsPathEnd → SucceedHandleSignal → waitForTaskToken; signal ingress (e.g. API Gateway → Lambda → SendTaskSuccess) looks up the token by (executionId, signalName) in an external storeHandleInterruption → Parallel where the interruption branch waits on its signal, runs the handler, then deliberately fails; ASL cancels the main branch; a Catch routes to the continuationEmbedded → recursive compile of the inner WIO
Out of scope (v1)
FlatMap is forbidden. Its continuation is Out => WIO[...] — the workflow shape depends on a runtime value and can't be statically compiled to a fixed ASL graph. Rewrite as Fork / AndThen.
Tradeoffs accepted
- State lives in ASL JSON, passed snapshot-in / snapshot-out between Tasks. This is the closest match to native Step Functions and gives free per-step state visibility in the console. Consequence: the event-sourced runtime model is dropped for this backend — no separate event log; constructs that lean on it (checkpoint/replay) don't carry over.
- ASL has a 256KB payload size limit between transitions; large accumulated state would need externalization.
- All
WCState and signal/event types need JSON codecs.
Open questions for the prototype
- Stable task IDs across deploys — start from existing names, fallback to auto-generated anonymous ids.
- Token store backend for
waitForTaskToken — DynamoDB is conventional, but the prototype can pick anything supporting (executionId, signalName) → token.
This is a rough sketch of the idea. I never worked with ASF, so raising this mostly to gather feedback and highlight a possibility.
Idea
Compile a
WIOprogram to an AWS Step Functions state machine (ASL JSON). The Scala app is deployed as a single "fat" Lambda that hosts a dispatch table for every Scala function in the WIO tree. ASL handles control flow and durability; the Lambda handles all computation.High-level shape
The compiler walks a
WIOand produces:Map[TaskId, ...]— every WIO node holding a Scala function gets a stable ID and a slot. This coversRunIO,Pure,Transform,Forkpredicates, dynamicTimerdurations,Loopconditions,Parallelcombiners — effectively every node that isn't pure structure.At runtime, every ASL Task state invokes the same Lambda with
{taskId, state}. The Lambda looks uptaskId, runs the Scala function, and returns a new state.Node mapping
(Approximate, to be verified)
RunIO/Pure/Transform→ Task (Lambda dispatch)AndThen→NextFork→ Task computing a discriminator, then Choice on it. (ASL Choice is restricted comparisons over JSONPath and can't express arbitrary Scala predicates; routing through the Lambda keeps this general.)Loop→ body subgraph + Choice + back-edgeParallel→ ASLParallelwith a Lambda combinerHandleError/HandleErrorWith→CatchRetry→ ASLRetrydirectivesTimerstatic →Wait; dynamic → Lambda computes duration →WaitwithSecondsPathEnd→SucceedHandleSignal→waitForTaskToken; signal ingress (e.g. API Gateway → Lambda →SendTaskSuccess) looks up the token by(executionId, signalName)in an external storeHandleInterruption→Parallelwhere the interruption branch waits on its signal, runs the handler, then deliberately fails; ASL cancels the main branch; aCatchroutes to the continuationEmbedded→ recursive compile of the inner WIOOut of scope (v1)
FlatMapis forbidden. Its continuation isOut => WIO[...]— the workflow shape depends on a runtime value and can't be statically compiled to a fixed ASL graph. Rewrite asFork/AndThen.Tradeoffs accepted
WCStateand signal/event types need JSON codecs.Open questions for the prototype
waitForTaskToken— DynamoDB is conventional, but the prototype can pick anything supporting(executionId, signalName) → token.