Option to stop a config without killing (or extend amount of time before SIGKILL is sent)

[reidsunderland]

In some cases, for example when shutting down a cluster for patching or when changing the settings of a config, it would be nice to have the option to stop configs without killing processes. i.e. send SIGTERM and wait forever, rather than waiting a while then sending SIGKILL.

I think the easiest implementation would be adding a command line argument that lets us increase the number of attempts before SIGKILL is sent.

In most cases, this would allow the process to finish sending its current batch of files before shutting down.

Semi-related: #618 - that issue is mainly talking about not abandoning retries when scaling down instances.

[reidsunderland]

If we choose to allow the number of attempts to be increased, then we should also set an upper limit on the time to wait:

sarracenia/sarracenia/sr.py

Line 2487 in dabd5ed

ttw = 1 << attempts

[petersilva]

In traditional linux/unix daemon init system vernacular, this type of operation is usually called graceful 👍

  /etc/init.d/sr3 graceful

apache implements this with that name also.

What are the things we want to wait for?

• resends? how many some of them go on for days?

• as a separate measure, on receipt of TERM, we could also set "messageCountMax" to 1 so that the next time it reads a batch, it will only read one message and exit after a while.

hmm....

[petersilva]

We could implement graceful as the same as stop, but it just never sends the SIGKILL at all, so you would use stop when the system is going down, so there is no point to leaving stuff running, but graceful would do all the same processing, but just never send the final SIGKILL.

The downside of an ignorant graceful is that it will not wait for everything to be stopped before returning, but at least it doesn't kill stuff in a non-recoverable/data loss potential way.

So instead, I'm building a branch to do two things:

• I propose to lengthens the time it waits to what is appropriate for the running flows, based on average batch processing times in the past.
• I propose to complete only the current transfer in progress, and put the rest of the batch into the retry queue, so that the process exits more quickly.

Not sure if it is enough, but it would be an improvement.

It might be worth describing how stop works now.

How stop currently works:

• receive a TERM signal, note that we should stop.
• complete normal processing of the current batch.
• try posting some retries.
• close cleanly.

Exit conditions that can make the stop take a long time:

• large batch settings (so a large number of transfers to get through before exit.)
• even with a small batch, really large files might be in transit for a long time.
• if connections are broken, trying to re-connect might hang
  • the moth class at least, has checks in connection estab. for _stop_requested, and will not hang trying to re-connect.
  • a connection attempt started before the stop was issued may require minutes to timeout, and the timeout might mask/prevent TERM from being processed properly. Additional testing needed.

Measure Average Batch Processing Time, and Tune sleep Between Signals Accordingly:

The following is intended to make the interval betwee sending of SIGTERM adapt to the actual duration
of batch processing. Since the duration should be a minimum of 1/2 the batch processing time, and the attempts is 5, this should mean the stop will wait at least 2.5 complete average batch processing times before it moves on to SIGKILL.

• If the batches are processed really quickly, then it should exit quickly.
• If the batches take long to process, sr3 stop will wait a reasonable amount of time (but that could be quite long.)

implementation:

• Each instance could write a tmp file, or a metric, that indicates the mean time to process a batch. (updated after each batch is processed.)
• sr3 status display could include the average batch processing time in the full display.
• When stop is run:
  • Look at that file for each configuration to be run, and find the maximum.
  • wait 1/2 the maximum time derived above (or the old ttw whichever is greater) as the time to wait before the next signal is sent.
• It also notes which of the components has the longest batch processing time.

Complete current transfer, but put the rest of them in the retry queue.

Abort after current transfer, rather than continuing the current batch.

• after each transfer, check if stop_requested in the do() / action phase.
  • if so, then append the message to failed so that it goes into the retry_queue, then next message.

Notes:

current algo in more detail (used to derive above):

• stop sends a TERM signal (aka signal 15) to each instance.
• the instances have a signal handler for SIGTERM ( in the classes, something like _stop_signal, or _active_stop_signal. There may be some auditing required to figure out whether SIGTERM is properly caught in all necessary situations.
• Assuming the signal is caught by a running instance, that means "stop_signal" runs, which then calls the flow/init.py stop_request().
• flow's stop_request then sets flow._stop_requested=True runs the "please_stop" plugins.
  • in moth/init.py please_stop, sets moth._stop_requested=True
• the flow continues processing the current batch of messages.
• At the end of the batch, if a _stop_requested is set, it won't sleep but loop immediately to the next cycle.
• At the start of the cycle, it checks if _stop_requested, and if so, it sets stopping to True, meaning this is the last pass through the loop.
• because stopping is set, it skips the gather() phase (meaning no new retries will be loaded either.)
  * note that the moth getNewMessage routines don't check for _stop_requested and return, which, in a belt and suspenders kind of way, they probably should.
• so then it runs what should be an empty: len(self.worklist.incoming) == 0
  *causes self.spamming to be set to True, and last gather_len==0
• filter() is called normally:
  • since the incoming worklist is empty, filter does nothing.
• work is called normally:
  • since the incoming worklist is empty work does nothing.
• post is called normally, except in a poll where last gather where the worklist is thinned to contain only retries.
  • it's hard to see anything useful happen here, because the worklist is nill from the beginning.
• It finishes the loop, again...
• at the start of the loop: it sees it was already stopping, so it calls self.close()
• self.close() runs all on_stop plugins.
  • flowcb/retry/ on_stop persists retry queues to disk. (by calling close.)

[reidsunderland]

This all sounds good. The only thing I'm a little bit hesitant about is putting messages in the retry queue. When using graceful to shutdown processes when removing a node from the cluster, those files would be stuck in the retry queue until the node is back, which is often at least an hour, sometimes longer if there's a hardware issue.

I also imagine graceful would be used when reducing instances, and in that case we also wouldn't want messages in the retry queue.

When we kill instances, RabbitMQ detects that the messages have not been acked and they get requeued, so another node can process those files instead.

[petersilva]

I guess it's a question of what are we trying to accomplish? Why do we care when we send SIGKILL?

My guess is that when we send SIGKILL, we are worried about something undefined happenning and transfers being dropped. If that's the problem, we can address it two ways:

a) make sure everything in progress completes, or

b) make sure everything not done gets queued somewhere.

The problem with option a) is that there can be a lot of stuff in flight (say batch 500) and so having them all drain might take a long time, especially if the reason we are shutting down is that the other end is broken, so each item in the batch needs to timeout, so on average 250 timeouts before we get to a sane state.

There is no timely way to deal with retries, it could be days, or it could never recover, and we just eventually give up.
It doesn't seem reasonable to try to complete everything in flight.

so b) is the only thing we can do. Right now we have two implementations of retry queues: disk, which is local, and redis, which is shared and does not have this problem. There could be a third one that uses nack, rather than queueing anything, except that only works on subscribers.

I think the problem of shutdowns to change topology is something to address as in the individual retry queue implementations. I think that is something specific to the disk implementation, and like it is enough to address it with #618 itself.

[petersilva]

otoh, we could implement a specific command that we know will take a very long time, with the express intent of some kind of topology change.... that would be sr3 drain as proposed in #895.

[petersilva]

So there are two different cases where we want to stop:

• to avoid using SIGKILL on stuff that hasn't cleanly stopped.

  • stop more quickly so SIGKILL doesn't have to be used (one PR stop faster by queuing rest of batch for retry. #1551.)
  • wait an longer before issuing SIGKILL by knowing how long a given client does transfers. (the other PR match wait time between SIGTERM to 1/2 a batch. #1552 )

• to drain a configuration so all pending transfers complete cleanly:

  • like sr3 drain?  #895 sr3 drain , or some kind of graceful command.
  • These are not really stop commands... they can be much, much longer than would be acceptable ... it is a different operation.

The two cases are different, both can be addressed, but likely quite differently.
I don't think they contradict each other, unless one forces them both into stop ...