multi-client: failover enhancements

[iurii-ssv]

This PR is meant to improve multi-client failover capabilities, previous implementation is using a simple heuristic of "client being synced" as the best way to predict if the client can serve API request or not (aka "active" client) + deactivating client under certain failure scenarios during API request execution, the issue is - that behavior doesn't cover some of the cases we've encountered in practice (eg. fetch-duties call was just timing out due to ~12s deadline we set, but at the same time we weren't trying to switch to another client presumably because the one we were trying reported it's synced).

Client being synced doesn't guarantee it's gonna serve API requests faithfully in a stable manner, hence a better heuristic is needed. A simple scoring system implemented in this PR (as seamlessly as possible on top of the previous sync-based approach to keep the code well-aligned with upstream) would do a better job at differentiating between good & bad clients, now failover works roughly like this:

• every client is classified into two categories: active/inactive based on its sync status (checked in background periodically)
• only currently active clients can be chosen to process API requests
• all clients (active or deactivated) have a score assigned to them, that score reflects how well they are expected to serve API requests based on the past history, there is no differentiation between various API endpoints (eg. failing endpoint1 call has the same effect on client score as failing endpoint2 call)
• the client with the highest score always serves API requests

Besides the added scoring mechanism this PR is also addressing the following edge-cases:

• whenever a client request times out with context.DeadlineExceeded error the previous implementation will simply keep trying that very same client every time without any failover, this is not ideal because this can very well happen regardless of what timeout(deadline) value the caller chooses due to variety of reasons: this particular client being overloaded at the moment, temporary network issue or crash while sync status hasn't updated yet, etc. - there is no reason not to try another client in that scenario
• (could potentially modify this behavior) additionally, whenever a client returns 4** type error response the previous implementation will simply keep trying that very same client every time without any failover - while the request itself is the most likely problem in this case, there is also a chance a bug in the client is the root cause for the request failing - so I'm not 100% we want to "trust" single client response (in SSV setting it would probably be better to fail over to another client just in case it's a client-bug, but I'm not changing this behavior for now)

I think this change should also remove the need for implementing things like ssvlabs/ssv#2339 (@nkryuchkov do you think we should partially-revert stuff added in ssvlabs/ssv#2339 ?).

Note, this PR is complementary to attestantio#192 but instead of "trying to make sync reporting more reliable" it targets the failover algorithm itself to make it more generic & robust.

[iurii-ssv]

@greptileai could you review this PR for me pls ?

[nkryuchkov]

good job, let's make sure this is tested well enough with real nodes