Skip to content

Fix behavior of --wait when used together with --agent - #78

Merged
LudovicoRighi merged 11 commits into
mainfrom
fix-wait-option
Aug 15, 2025
Merged

Fix behavior of --wait when used together with --agent#78
LudovicoRighi merged 11 commits into
mainfrom
fix-wait-option

Conversation

@LudovicoRighi

@LudovicoRighi LudovicoRighi commented Aug 12, 2025

Copy link
Copy Markdown
Collaborator

Currently, using --wait with --agent does not work, i.e. a Procstar agent would connect to Apsis but:

  • Apsis would not schedule any runs on that agent (because the agent would transition immediately to the ShutdownState.done and so it would not be available for use)
  • and at the same time the agent would not even exit

After this change, when using --wait and --agent the Procstar agent:

  • connects and waits until a single run will be executed on it
  • right after the first run is started, it transitions to the ShutdownState.idling so not to accept any more runs
  • when the run terminates and the process is deleted, it shuts down.

I've tested this with a local Apsis and Procstar.

@LudovicoRighi
LudovicoRighi force-pushed the fix-wait-option branch 6 times, most recently from 4f59925 to f691a92 Compare August 12, 2025 18:06

@gusostow gusostow left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Congrats on the Rust! This seems like it could take a little more work to get right. But this should be good enough to start testing with even if it's not released.

Comment thread src/main.rs Outdated
// Wait for work to be assigned or shutdown signal
tokio::select! {
_ = procs.wait_for_shutdown() => return,
_ = tokio::time::sleep(tokio::time::Duration::from_millis(100)) => {},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should use the procs notification mechanism instead of polling

Comment thread src/main.rs Outdated
// Ready to shut down now.
procs.set_shutdown(shutdown::State::Done);
// Work has been assigned! Set shutdown to Idling to prevent accepting more work
procs.set_shutdown(shutdown::State::Idling);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to worry about race condition here where proc start request comes in before we start idling?

@LudovicoRighi LudovicoRighi left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Congrats on the Rust! This seems like it could take a little more work to get right.

Thanks! Yeah the PR was still work in progress, but I appreciated the early feedback since it was your last day before holidays.

But this should be good enough to start testing with even if it's not released.

I'd actually like to get this released so that it's easier for me to test within containers running on EC2 instances. I'll ask Oleh to review.

Do we need to worry about race condition here where proc start request comes in before we start idling?

I've actually decided to not limit the Procstar agent to accept a single process when using the --wait mode. This seems more in line with Alex original design and also, for what concerns the Apsis on ECS project, we will enforce a single run on each Procstar through a unique group_id (or connection_id). Therefore there's probably no need to make --wait that strict; in case, we could also do that change later on.

I've also added a configurable timeout that will make the agent to shutdown in case no processes are assigned within that time window.

@LudovicoRighi
LudovicoRighi marked this pull request as ready for review August 13, 2025 15:27
@LudovicoRighi
LudovicoRighi merged commit c839321 into main Aug 15, 2025
1 check passed
@LudovicoRighi
LudovicoRighi deleted the fix-wait-option branch August 15, 2025 12:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants