Cooperation with safely

[piotr-yuxuan]

Hello 👋 I haven't used this project but it looks great. While reading the readme, I wondered what you would think about using salutem in conjunction with com.brunobonacci/safely for example to integrate with the retry backoff.

[tobyclemson]

Hi @piotr-yuxuan,

Thanks for the pointer to com.brunobonacci/safely, I think it may be useful in check function implementations.

Currently, I am undecided as to whether salutem should be responsible for implementing backoff or whether that should be left up to the implementer of checks. I'm currently more of the opinion that health checks should continue to run even if a dependency is unhealthy so that any monitoring that depends on those health checks knows as soon as possible that the dependency has recovered.

The circuit breaker pattern is more important when trying to prevent a "cascading failure" where a large and ever growing volume of requests become blocked right the way back up through a call tree potentially involving and potentially overloading many components. In the case of salutem, the number of requests to dependencies is tightly controlled based on refresh intervals and timeouts such that the "cascading failure" problem isn't really a problem. As such, I haven't incorporated circuit breaking logic as yet as I'd like to get some more examples of the system in use.

I do however incorporate timeouts into all of the check functions I write, since even with the tight control over the number of requests produced by salutem, long running requests are still a bad practice and if they can be avoided, they should be.

How were you thinking safely should be incorporated?

Thanks,
Toby

[piotr-yuxuan]

I think we could use it inside a check-fn with :salutem/time-to-re-evaluation:

(fn [context callback-fn]
  (safely
    ;; IO call goes here, for example `http/get`.

    :on-error
    :max-retries 5
    :circuit-breaker :external-service))

Then, elsewhere in code we can reuse this circuit breaker:

(safely
  ;; IO call goes here, for example `http/get`.

  :on-error
  :max-retries 5
  :circuit-breaker :external-service)