Handle partial volume creation failures - #527
Open
Michelle Au (msau42) wants to merge 3 commits into
Open
Conversation
Michelle Au (msau42)
force-pushed
the
handle-create-volume-failure
branch
5 times, most recently
from
July 28, 2026 03:29
3df47e1 to
958b436
Compare
Michelle Au (msau42)
changed the title
Collaborator
Author
|
I created new commits to address the review comments. I can squash before merge |
Michelle Au (msau42)
changed the title
Michelle Au (msau42)
force-pushed
the
handle-create-volume-failure
branch
2 times, most recently
from
July 29, 2026 00:26
ee1904e to
42bc74f
Compare
1. During CreateActor, the volumes to be created get instantiated in the Actor API in CREATING status. The Actor is still instantiated as SUSPENDED. However, the CreateVolume operation is moved to the beginning of ResumeActor. This is because CreateActor is intentionally designed to not be idempotent. The downside is that the user won't know dependent resource creation failed until the first resume. 2. Changes volume id to use actor uid instead of atespace+actorname, and add "substrate" prefix 3. Adds a new DELETING status that gets persisted before we start to delete volumes. This shifts the existing DB DELETE precondition checks to the new DELETING status. And now you can only delete the actor in DB from DELETING status. In the future, we will also add a workflow that will do mandatory node cleanup of volumes before starting to delete the volumes.
The steps are: - LoadActorForDeleteStep: Loads the actor state from store. - MarkDeletingStep: Validates state prerequisites (STATUS_SUSPENDED, STATUS_CRASHED, STATUS_DELETING) and updates actor and external volume statuses to STATUS_DELETING / ExternalVolume_DELETING. - DeleteVolumesStep: Deletes associated external volumes via deleteActorVolumes. - FinalizeDeletedStep: Deletes the actor from store and returns the deleted object.
Michelle Au (msau42)
force-pushed
the
handle-create-volume-failure
branch
from
July 29, 2026 00:33
42bc74f to
be6d8f9
Compare
| message ExternalVolume { | ||
| // actor_id + the volume name specified in the actor template. | ||
| string actor_volume_id = 1; | ||
| // Name of the volume specified in the actor template. |
Collaborator
There was a problem hiding this comment.
We should add validation for these fields. If you do it in a follow up maybe add a TODO comment in create_actor.go?
| PROVISIONING = 0; | ||
| CREATED = 1; | ||
| DELETING = 2; | ||
| STATUS_UNKNOWN = 0; |
Collaborator
There was a problem hiding this comment.
Nit: This should be STATUS_UNSPECIFIED.
| DELETING = 2; | ||
| STATUS_UNKNOWN = 0; | ||
| // Volume creation pending in the storage system. | ||
| PENDING = 1; |
Collaborator
There was a problem hiding this comment.
Let's add STATUS_ prefix here and on the other values below for consistency with other enums.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #232
However, the CreateVolume operation is moved to the beginning of ResumeActor. This is because CreateActor is intentionally designed to not be idempotent. The downside is that the user won't know dependent resource creation failed until the first resume.
Changes volume id to use actor uid instead of atespace+actorname and adds "substrate" prefix
Adds a new DELETING status that gets persisted before we start to delete volumes.
This shifts the existing DB DELETE precondition checks to the new DELETING status. And now you can only delete the actor in DB from DELETING status.
In the future, we will also add a workflow that will do mandatory node cleanup of volumes before starting to delete the volumes.