Wire workflow reset to support CHASM Nexus operations (probe-and-fallback)#10854
Open
tekkaya wants to merge 2 commits into
Open
Wire workflow reset to support CHASM Nexus operations (probe-and-fallback)#10854tekkaya wants to merge 2 commits into
tekkaya wants to merge 2 commits into
Conversation
tekkaya
force-pushed
the
gokhan/nexus-reset-chasm-fallback-draft
branch
13 times, most recently
from
c565888 to
fa729a9
Compare
tekkaya
changed the title
tekkaya
force-pushed
the
gokhan/nexus-reset-chasm-fallback-draft
branch
3 times, most recently
from
5bbda34 to
fa644ff
Compare
tekkaya
added a commit
that referenced
this pull request
Jul 2, 2026
Now that #10854 wires the CHASM reset/reapply dispatch, use the replay context (from the previous commit) at the two sites that re-run CHASM transitions from history: - mutable_state_rebuilder: base rebuild (def.Apply) - workflow_resetter.cherryPickChasmEvent: reset reapply (def.CherryPick) Add MutableState.ChasmWorkflowComponentForReplay, which builds the workflow component context via chasm.NewMutableContextForReplay so metrics emitted inside transitions are suppressed while re-applying events (avoiding double-counting caller-side Nexus metrics). ChasmWorkflowComponent (live callers) is unchanged; both share a private helper. Update the reset/rebuilder mock expectations accordingly.
5 tasks
…tion
Nexus operations can be backed by either the HSM tree or the CHASM tree,
and both coexist on the same mutable state. Before this change,
reset/rebuild/reapply dispatched Nexus events only through HSM, and async
completion committed to one framework by token type. So once
nexusoperation.enableChasmWorkflowOperations is on, CHASM-backed ops were
silently not rebuilt on reset, and an op that "moved" trees via reset could
not be completed.
This wires reset, state rebuild, the event reapplier, and async completion
to support either tree using a probe-and-fallback approach keyed by the
cross-tree-stable ScheduledEventId (HSM node ID; CHASM Workflow.Operations
map key). No new event field/marker.
- Reset reapply (ndc/workflow_resetter.go): reapplyEvents probes HSM, then
falls back to the CHASM workflow registry, with an explicit
cherryPickOutcome (applied/skipped/fallback) to distinguish "recognized
but not cherry-pickable" (skip) from "not owned here" (fallback).
- Flag-driven rebuild-create (workflow/mutable_state_rebuilder.go): the
NexusOperationScheduled create event is routed by the per-namespace
enableChasmWorkflowOperations flag (configs.Config
EnableChasmNexusWorkflowOperations); on CHASM create error it falls back
to HSM. Non-create events use HSM-first / CHASM-fallback.
- Completion-token fallback, both directions, server-side in History
(service/history/handler.go):
- HSM token -> CHASM op: on HSM NotFound, recover ScheduledEventId from
the StateMachineRef and complete the op on the current run's CHASM tree.
- CHASM token -> after-reset re-resolution: a CHASM token pins the
schedule-time run via its ChasmComponentRef; a reset closes that run and
rebuilds the op on a new current run. On failure against the pinned run,
re-resolve the op by ScheduledEventId on the current run and complete it
in CHASM or fall back to the HSM handler depending on where the rebuild
placed it. Namespace/business IDs are read from the ChasmComponentRef,
not the legacy NamespaceId/WorkflowId completion fields.
- Fallback gating keeps invalid/unrelated tokens rejected rather than
silently re-targeting the current run: archetype must be the Workflow
archetype; the pinned run must actually exist (a reset closes it but
leaves it readable, whereas a bogus run id does not resolve) -- a
positive existence check, not error-type based; and namespace EnableChasm
must hold (avoids a noopChasmTree). The frontend completion handler stays
a pure router by token type.
- Event reapplier (ndc/events_reapplier.go): plumbs the real CHASM workflow
registry through EventsReapplierImpl so the replication / current-run
reapply path gets the same fallback.
- Plumbing: new ShardContext.ChasmWorkflowRegistry() accessor (interface,
ContextImpl, factory, regenerated mock, test util), fx-auto-wired.
Tests:
- Unit: handler_test.go (token/ref helpers, completion conversion);
mutable_state_rebuilder_test.go (flag-on CHASM-create-error -> HSM).
- Functional (tests/nexus_workflow_test.go):
TestNexusOperationResetCrossTreeCompletion (op created in one tree, flag
flipped, reset rebuilds into the other tree and the completion reapplies
across trees, both directions); TestNexusOperationCompletionNotFoundInEitherTree
(tampered token -> NotFound preserved); TestNexusOperationAsyncCompletionAfterReset
now enabled for the CHASM variant (live completion after reset resolves
onto the reset-created current run).
Two fixes to the server-side completion-token fallback: - Gate the HSM-token -> CHASM-op fallback (completeNexusOperationChasmFallback) on the token's schedule-time run actually existing, mirroring the CHASM-token path. Previously it re-targeted the current run using only ns/workflow/ scheduledEventId, so a bogus or tampered run ID with an otherwise valid event/request ID could complete the current-run op instead of preserving NotFound. - Preserve the canceled outcome in the CHASM-token -> HSM-op fallback. CompleteNexusOperationChasmRequest carries no explicit outcome state, so nexusOperationErrorFromChasmRequest defaulted non-OperationError failures to Failed. It now infers Canceled from CanceledFailureInfo, matching the native CHASM path (Operation.HandleNexusCompletion), so a completion records the same terminal state (canceled vs failed) regardless of which tree the reset placed the op in.
tekkaya
force-pushed
the
gokhan/nexus-reset-chasm-fallback-draft
branch
from
11f5712 to
2e7ae0f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed?
Makes workflow reset, state rebuild, the event reapplier, and async completion work for Nexus operations that live in either the HSM tree or the CHASM tree, using a probe-and-fallback keyed by the cross-tree-stable
ScheduledEventId. No new event field/marker.workflow/mutable_state_rebuilder.go): theNexusOperationScheduledcreate event is routed by the per-namespacenexusoperation.enableChasmWorkflowOperationsflag (CHASM when on, HSM when off), falling back to HSM on CHASM create errors. Non-create events use HSM-first / CHASM-fallback.ndc/workflow_resetter.go) and the event reapplier (ndc/events_reapplier.go): same HSM↔CHASM fallback, plumbed the CHASM workflow registry through both.service/history/handler.go): an HSM-token completion whose op now lives in CHASM completes via the CHASM tree; a CHASM-token completion after a reset re-resolves the op on the current run and completes it in whichever tree the rebuild placed it (CHASM or HSM). Gated so unrelated tokens still return NotFound (Workflow archetype only, schedule-time run must exist, namespaceEnableChasm).service/history/statemachine_environment.go): a StateMachineRef reconstructed for the CHASM-token→HSM-op completion carries no versioned-transition info, which the HSM ref validation dereferenced. Made the staleness check nil-safe (unchanged for normal tokens, which always populate it).ShardContext.ChasmWorkflowRegistry()accessor (interface + impl + factory + regenerated mock + test util), fx-wired.Why?
Nexus operations can be backed by either the HSM or CHASM tree, and both coexist on the same mutable state. Before this change, reset/rebuild/reapply dispatched Nexus events only through HSM, and completion committed to one framework by token type — so once
enableChasmWorkflowOperationsis on, CHASM-backed ops were silently not rebuilt on reset, and an op that "moved" trees via reset couldn't be completed. The probe-and-fallback approach lets both frameworks coexist and resolve an op correctly regardless of which tree currently owns it.How did you test it?
TestNexusOperationAsyncCompletionHSM+CHASM, incl. token-validation cases that guard the fallback gating)handler_test.gotoken/ref/conversion helpers;mutable_state_rebuilder_test.gocreate-routing + non-create HSM↔CHASM dispatch)TestNexusOperationResetCrossTreeCompletion— op created in one tree, flag flipped, reset rebuilds it into the other tree and reapplies the completion across trees.TestNexusOperationCrossTreeCompletionAfterReset— live completion after a tree-moving reset, both directions (HSM-token→CHASM-op, CHASM-token→HSM-op), withDescribeMutableStateasserting the op actually moved trees.TestNexusOperationCompletionNotFoundInEitherTree— tampered token → NotFound preserved.TestNexusOperationAsyncCompletionAfterReset— now enabled for CHASM (same-tree, current-run resolution after reset).